[Starlink] Starlink hidden buffers

[Ulrich Speidel]

On 15/05/2023 3:33 pm, David Lang wrote:
> On Mon, 15 May 2023, Ulrich Speidel wrote:
>
> > On 14/05/2023 9:00 pm, David Lang wrote:
> >> On Sun, 14 May 2023, Ulrich Speidel wrote:
> >>
> >> >> I just discovered that someone is manufacturing an adapter so 
> you no
> >> >> longer have to cut the cable
> >> >>
> >> >>
> >> 
> https://www.amazon.com/YAOSHENG-Rectangular-Adapter-Connect-Injector/dp/B0BYJTHX4P 
> <https://www.amazon.com/YAOSHENG-Rectangular-Adapter-Connect-Injector/dp/B0BYJTHX4P>
>
> >> >>
> >> > I'll see whether I can get hold of one of these. Cutting a cable 
> on a
> >> > university IT asset as an academic is not allowed here, except if it
> >> > doesn't meet electrical safety standards.
OK, we have one on order, along with PoE injector and power supply. 
Don't hold your breath, though, I'll be out of the country when it 
arrives and it'll be late July before I get to play with it.
> >> >
> >> > Alternatively, has anyone tried the standard Starlink Ethernet 
> adapter with
> >> > a PoE injector instead of the WiFi box? The adapter above seems 
> to be like
> >> > the Starlink one (which also inserts into the cable between Dishy 
> and
> >> > router).
> >>
> >> that connects you a 2nd ethernet port on the router, not on the dishy
> >>
> >> I just ordered one of those adapters, it will take a few weeks to 
> arrive.
> > How do we know that the Amazon version doesn't do the same?
>
> because it doesn't involve the router at all. It allows you to replace 
> the
> router with anything you want.
>
> People have documented how to cut the cable and crimp on a RJ45 
> connector, use a
> standard PoE injector, and connect to any router you want. I was 
> preparing to do
> that (and probably still will for one cable to use a different 
> locations to
> avoid having a 75 ft cable from the dish mounted on the roof of my van 
> to the
> router a couple feet away), This appears to allow me to do the same 
> functional
> thing, but without cutting the cable.
Let's see whether they actually work any different ;-) They're sure in 
the same position in the cable.
>
> >> >> > I suspect they're handing over whole cells, not individual 
> users, at a
> >> >> > time.
> >> >>
> >> >> I would guess the same (remember, in spite of them having 
> launched >4000
> >> >> satellites, this is still the early days, with the network 
> changing as
> >> >> more launching)
> >> >>
> >> >> We've seen that it seems that there is only one satellite 
> serving any cell
> >> >> one time.
> >>
> >> > But the reverse is almost certainly not true: Each satellite must 
> serve
> >> > multiple cells.
> >>
> >> true, but while the satellite over a given area will change, the 
> usage in
> >> that area isn't changing that much
>
> > Exactly. But your underlying queue sits on the satellite, not in the 
> area.
>
> only if the satellite is where you have more input than output. That 
> may be the
> case for users uploading, but for users downloading, I would expect 
> that the
> bandwidth bottleneck would be from the Internet connected ground 
> station to the
> satellite, with the satellite serving many cells but only having one 
> uplink.

Leaving lasers aside for the moment.

I'd expect there to be one queue for each satellite uplink at the 
gateway ground station, and that the occupancy of that queue depends on 
how much demand the users on that satellite currently produce. So as a 
remote terminal switches satellites, even if the ground station remains 
the same, it sees different queuing delays for its inbound traffic at 
the ground station.

For the uplink from the user terminal, we can't have multiple users 
accessing the same uplink channel (however you define "channel" - 
frequency, time slot, spreading code, beam, polarisation, any 
combination thereof, ...) simultaneously as they are not able to 
coordinate and you wouldn't want random access for your main data link 
channel because of the hidden node collisions this would produce (a 
random access channel paired with an access grant channel is a different 
story). So you'd get slot assignments from the satellite obviously, and 
the queue for one of these sits at the user terminal. But what isn't 
clear to me is whether the satellites are truly only handled by a single 
ground station at a time, or perhaps by multiple ground stations. If 
it's the latter, then you might end up with a situation where you have 
more traffic arriving at the satellite than it can dispatch to its 
ground station(s), and then you'd need a queue in the uplink direction 
also.

Similarly, if the combined uplinks from the ground station are able to 
deliver more data than the satellite can downlink to its users through 
its current slot assignments, we need a queuing system on the satellite 
in that direction, too.

Add lasers in, and it seems like having some sort of buffer on the 
satellites is a must.

>
> >> >> But remember that the system does know how much usage there is 
> in the cell
> >> >> before they do the handoff. It's unknown if they do anything 
> with that, or
> >> >> if they are just relaying based on geography. We also don't know 
> what the
> >> >> bandwidth to the ground stations is compared to the dishy.
> >>
> >> > Well, we do know for NZ, sort of, based on the licences Starlink 
> has here.
> >>
> >> what is the ground station bandwith?
> >
> > https://rrf.rsm.govt.nz/ui/search/licence 
> <https://rrf.rsm.govt.nz/ui/search/licence> 
> - seach for "Starlink"
> >
> > ...all NZ licences in all their glory. Looking at Starlink SES 
> (satellite
> > earth station) TX (which is the interesting direction I guess):
> >
> > - Awarua, Puwera, Hinds, Clevedon, Cromwell, Te Hana: 29750.000000 
> TX (BW =
> > 500 MHz)
> > - Awarua, Puwera, Hinds, Clevedon, Cromwell, Te Hana: 28850.000000 
> TX (BW =
> > 500 MHz)
> > - Awarua, Puwera, Hinds, Clevedon, Cromwell, Te Hana: 28350.000000 
> TX (BW =
> > 500 MHz)
> > - Awarua, Puwera, Hinds, Clevedon, Cromwell, Te Hana: 28250.000000 
> TX (BW =
> > 500 MHz)
> > - Awarua, Puwera, Hinds, Clevedon, Cromwell, Te Hana: 27750.000000 
> TX (BW =
> > 500 MHz)
> >
> > So 2.5 GHz up, licensed from 6 ground stations. Now I'm not 
> convinced that
> > they would use all of those from all locations simultaneously 
> because of the
> > risk of off-beam interference. They'll all be transmitting south, 
> ballpark.
> > If there was full re-use at all ground stations, we'd be looking at 
> 15 GHz.
> > If they are able to re-use on all antennas at each ground station, 
> then we're
> > looking at 9 golf balls each in Puwera, Te Hana, Clevedon, Hinds and
> > Cromwell, and an unknown number at Awarua. Assuming 9 there, we'd be 
> looking
> > at 135 GHz all up max.
> >
> > Awarua and Cromwell are 175 km apart, Hinds another 220 km from 
> Cromwell,
> > then it's a hop of about 830 km to Clevedon, and from there another 
> 100 km to
> > Te Hana, which is another 53 km from Puwera, so keeping them all out 
> of each
> > other's hair all the time might be a bit difficult.
> >
> > Lots of other interesting info in the licenses, such as EIRP, in 
> case you're
> > wanting to do link budgets.
>
> I was asking more in terms of Gb/s rather than MHz of bandwidth. 
> Dedicated
> ground stations with bigger antennas, better filters, more processing and
> overall a much higher budget can get much better data rates out of a 
> given
> amount of bandwidth than the user end stations will.
>
> it's also possible (especially with bigger antennas) for one ground 
> station
> location to talk to multiple different satellites at once (the aiming 
> of the
> antennas can isolate the signals from each other)

Well, the Gb/s is what a link budget would give you if you knew the 
modulation scheme(s) and any FEC used. The ground station antennas are 
normal parabolic dishes in radomes for all we can tell, and are all the 
same size, so you can kind of estimate aperture and hence gain 
reasonably well. Path loss depends a little on distance and path quality 
(weather / rain fade), and we don't really know in how far their modems 
use adaptive rates to cope with this (my Ookla tests during Cyclone 
Gabrielle most certainly don't rule this out - rates went down both ways 
during the thick of it). I guess we know relatively little about the 
on-board phased array on the satellites (apart from very loose bounds), 
which restricts our ability to say much about gain there (and potential 
for spatial separation / re-use of beam frequencies). We also don't know 
how Starlink manages its frequency allocation across its ground stations.

It's certainly noticeable here that they seem to have sets of three 
grouped together in a relatively compact geographical area (you could 
visit all NZ North Island ground stations in a day by car from Auckland, 
Auckland traffic notwithstanding, and at a stretch could do the same 
down south from Hinds to Awarua if you manage to ignore the scenery, but 
getting from the southernmost North Island ground station to the 
northernmost South Island one is basically a two day drive plus ferry trip).

>
> >> As latency changes, figuring out if it's extra distance that must be
> >> traveled, or buffering is hard. does the latency stay roughly the 
> same until
> >> the next satellite change? or does it taper off?
>
> > Good question. You would expect step changes in physical latency 
> between
> > satellites, but also gradual change related to satellite movement. 
> Plus of
> > course any rubble thrown into any queue by something suddenly 
> turning up on
> > that path. Don't forget that it's not just cells now, we're also 
> talking up-
> > and downlink for the laser ISLs, at least in some places.
>
> how far do the satellites move in 15 min and what effect would that 
> have on
> latency (I would assume that most of the time, the satellites are 
> switched to as
> they are getting nearer the two stations, so most of the time, I would 
> expect a
> slight reduction in latency for ~7 min and then a slight increase for 
> ~7 min,
> but I would not expect that this would be a large variation

Dishy tracks most satellites for significantly less than 15 minutes, and 
for a relatively small part of their orbit. Let me explain:



This is an obstruction map obtained with starlink-grpc-tools 
(https://github.com/sparky8512/starlink-grpc-tools). The way to read 
this is in polar coordinates: The centre of the image is the dishy 
boresight (direction of surface normal), distance from the centre is 
elevation measured as an angle from the surface normal, and direction 
from the centre is essentially the azimuth - top is north, left is west, 
bottom is south, and right is east. The white tracks are the satellites 
dishy uses, and a graph like this gets built up over time, one track at 
a time. Notice how short the tracks are - they don't follow the 
satellite for long - typically under a minute. The red bits are 
satellites getting obscured by the edge of our roof.

I've also attached a time lapse movie of how one of these graphs builds 
up - if I correctly remember (the script is on another machine), one 
frame in the video corresponds to 5 seconds.

Conclusion: latency change from tracking one satellite is smaller than 
the latency difference as you jump between satellites. You could be 
looking at several 100 km of path difference here. In an instant. Even 
that, at 300,000 km/s of propagation speed, is only in the order of 
maybe 1 ms or so - peanuts compared to the RTTs in the dozens of ms that 
we're seeing. But if you get thrown from one queue onto another as you 
get handed over - what does that do to the remote TCP stack that's 
serving you?

>
> >> If it stays the same, I would suspect that you are actually hitting a
> >> different ground station and there is a VPN backhaul to your egress 
> point to
> >> the regular Internet (which doesn't support mobile IP addresses) 
> for that
> >> cycle. If it tapers off, then I could buy bufferbloat that gets 
> resolved as
> >> TCP backs off.
> >
> > Yes, quite sorting out which part of your latency is what is the 
> million
> > dollar question here...
> >
> > We saw significant RTT changes here during the recent cyclone over 
> periods of
> > several hours, and these came in steps (see below), with the initial 
> change
> > being a downward one. Averages are over 60 pings (the time scale 
> isn't 100%
> > true as we used "one ping, one second" timing) here.
> >
> >
> > We're still not sure whether to attribute this to load change or ground
> > station changes. There were a lot of power outages, especially in 
> Auckland's
> > lifestyle block belt, which teems with Starlink users, but all three 
> North
> > Island ground stations were also in areas affected by power outages 
> (although
> > the power companies concerned don't provide the level of detail to 
> establish
> > whether they were affected). It's also not clear what, if any, 
> backup power
> > arrangements they have). At ~25 ms, the step changes in RTT are too 
> large be
> > the result of a switch in ground stations, though, the path 
> differences just
> > aren't that large. You'd also expect a ground station outage to 
> result in
> > longer RTTs, not shorter ones, if you need to re-route via another 
> ground
> > station. One explanation might be users getting cut off if they 
> relied on one
> > particular ground station for bent pipe ops - but that would not 
> explain this
> > order of magnitude effect as I'd expect that number to be small. So 
> maybe
> > power outages at the user end after all. But that would then tell us 
> that
> > these are load-dependent queuing delays. Moreover, since those load 
> changes
> > wouldn't have involved the router at our site, we can conclude that 
> these are
> > queue sojourn times in the Starlink network.
>
> I have two starlink dishes in the southern california area, I'm going 
> to put
> one on the low-priority mobile plan shortly. These are primarily used 
> for backup
> communication, so I would be happy to add something to them to do latency
> monitoring. In looking at what geo-location reports my location as, I 
> see it
> wander up and down the west coast, from the Los Angeles area all the 
> way up to
> Canada.
Would be worthwhile to also do traceroutes to various places to see 
where you emerge from the satellite side of things.
>
> >> I think that active queue management on the sending side of the 
> bottleneck
> >> will handle it fairly well. It doesn't have to do calculations 
> based on what
> >> the bandwidth is, it just needs to know what it has pending to go out.
>
> > Understood - but your customer for AQM is the sending TCP client, 
> and there
> > are two questions here: (a) Does your AQM handle rapid load changes 
> and (b)
> > how do your TCP clients actually respond to your AQM's handling?
>
> AQM allocates the available bandwidth between different connections 
> (usually
> different users)
But it does this under the assumption that the vector for changes in 
bandwidth availability is the incoming traffic, which AQM gives 
(indirect) feedback to, right?
>
> When it does this indirectly for inbound traffic by delaying acks, the 
> results
> depend on the senders handling of these indirect signals that were never
> intended for this purpose.
>
> But when it does this directly on the sending side, it doesn't matter 
> what the
> senders want, their data WILL be managed to the priority/bandwidth 
> that the AQM
> sets, and eventually their feedback is dropped packets, which everyone 
> who is
> legitimate responds to. 

Understood. You build a control loop, where the latency is the delay in 
the control signal.

Classically, you have a physical bottleneck that the AQM manages, where 
the physical bandwidth doesn't change.

The available bandwidth changes, (mostly) as a result of TCP connections 
(or similarly behaved UDP applications) joining in slow start, or 
disappearing.

Basically, your queues grow and shrink one packet at a time.

Your control signal allows you (if they're well behaved) throttle / 
accelerate senders.

What you don't get are quantum jumps in queue occupancy, jump changes in 
underlying physical bandwidth, or a whole set of new senders that are 
completely oblivious to any of your previous control signals. But you 
get all that with satellite handovers like these.

So what if the response you elicit in this way is to a queue scenario 
that no longer applies?

> But even if they don't respond (say a ping flood or DoS
> attack), the AQM will limit the damage to that connection, allowing 
> the other
> connections trying to use that link to continue to function.
All understood.
>
> David Lang

-- 
****************************************************************
Dr. Ulrich Speidel

School of Computer Science

Room 303S.594 (City Campus)

The University of Auckland
u.speidel at auckland.ac.nz  
http://www.cs.auckland.ac.nz/~ulrich/
****************************************************************


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.bufferbloat.net/pipermail/starlink/attachments/20230525/ef367329/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: obstructions_pos2_762.png
Type: image/png
Size: 1289 bytes
Desc: not available
URL: <https://lists.bufferbloat.net/pipermail/starlink/attachments/20230525/ef367329/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: obs_map20-2-23.mp4
Type: video/mp4
Size: 169217 bytes
Desc: not available
URL: <https://lists.bufferbloat.net/pipermail/starlink/attachments/20230525/ef367329/attachment-0001.mp4>