[Starlink] Starlink hidden buffers

[David Lang]

On Mon, 15 May 2023, Ulrich Speidel wrote:

> On 14/05/2023 9:00 pm, David Lang wrote:
>> On Sun, 14 May 2023, Ulrich Speidel wrote:
>> 
>> >> I just discovered that someone is manufacturing an adapter so you no 
>> >> longer have to cut the cable
>> >>
>> >> 
>> https://www.amazon.com/YAOSHENG-Rectangular-Adapter-Connect-Injector/dp/B0BYJTHX4P

>> >>
>> > I'll see whether I can get hold of one of these. Cutting a cable on a 
>> > university IT asset as an academic is not allowed here, except if it 
>> > doesn't meet electrical safety standards.
>> >
>> > Alternatively, has anyone tried the standard Starlink Ethernet adapter with 
>> > a PoE injector instead of the WiFi box? The adapter above seems to be like 
>> > the Starlink one (which also inserts into the cable between Dishy and 
>> > router).
>> 
>> that connects you a 2nd ethernet port on the router, not on the dishy
>> 
>> I just ordered one of those adapters, it will take a few weeks to arrive.
> How do we know that the Amazon version doesn't do the same?

because it doesn't involve the router at all. It allows you to replace the 
router with anything you want.

People have documented how to cut the cable and crimp on a RJ45 connector, use a 
standard PoE injector, and connect to any router you want. I was preparing to do 
that (and probably still will for one cable to use a different locations to 
avoid having a 75 ft cable from the dish mounted on the roof of my van to the 
router a couple feet away), This appears to allow me to do the same functional 
thing, but without cutting the cable.

>> >> > I suspect they're handing over whole cells, not individual users, at a 
>> >> > time.
>> >>
>> >> I would guess the same (remember, in spite of them having launched >4000 
>> >> satellites, this is still the early days, with the network changing as 
>> >> more launching)
>> >>
>> >> We've seen that it seems that there is only one satellite serving any cell 
>> >> one time.
>> 
>> > But the reverse is almost certainly not true: Each satellite must serve 
>> > multiple cells.
>> 
>> true, but while the satellite over a given area will change, the usage in 
>> that area isn't changing that much

> Exactly. But your underlying queue sits on the satellite, not in the area.

only if the satellite is where you have more input than output. That may be the 
case for users uploading, but for users downloading, I would expect that the 
bandwidth bottleneck would be from the Internet connected ground station to the 
satellite, with the satellite serving many cells but only having one uplink.

>> >> But remember that the system does know how much usage there is in the cell 
>> >> before they do the handoff. It's unknown if they do anything with that, or 
>> >> if they are just relaying based on geography. We also don't know what the 
>> >> bandwidth to the ground stations is compared to the dishy.
>> 
>> > Well, we do know for NZ, sort of, based on the licences Starlink has here.
>> 
>> what is the ground station bandwith?
>
> https://rrf.rsm.govt.nz/ui/search/licence - seach for "Starlink"
>
> ...all NZ licences in all their glory. Looking at Starlink SES (satellite 
> earth station) TX (which is the interesting direction I guess):
>
> - Awarua, Puwera, Hinds, Clevedon, Cromwell, Te Hana: 29750.000000 TX (BW = 
> 500 MHz)
> - Awarua, Puwera, Hinds, Clevedon, Cromwell, Te Hana: 28850.000000 TX (BW = 
> 500 MHz)
> - Awarua, Puwera, Hinds, Clevedon, Cromwell, Te Hana: 28350.000000 TX (BW = 
> 500 MHz)
> - Awarua, Puwera, Hinds, Clevedon, Cromwell, Te Hana: 28250.000000 TX (BW = 
> 500 MHz)
> - Awarua, Puwera, Hinds, Clevedon, Cromwell, Te Hana: 27750.000000 TX (BW = 
> 500 MHz)
>
> So 2.5 GHz up, licensed from 6 ground stations. Now I'm not convinced that 
> they would use all of those from all locations simultaneously because of the 
> risk of off-beam interference. They'll all be transmitting south, ballpark. 
> If there was full re-use at all ground stations, we'd be looking at 15 GHz. 
> If they are able to re-use on all antennas at each ground station, then we're 
> looking at 9 golf balls each in Puwera, Te Hana, Clevedon, Hinds and 
> Cromwell, and an unknown number at Awarua. Assuming 9 there, we'd be looking 
> at 135 GHz all up max.
>
> Awarua and Cromwell are 175 km apart, Hinds another 220 km from Cromwell, 
> then it's a hop of about 830 km to Clevedon, and from there another 100 km to 
> Te Hana, which is another 53 km from Puwera, so keeping them all out of each 
> other's hair all the time might be a bit difficult.
>
> Lots of other interesting info in the licenses, such as EIRP, in case you're 
> wanting to do link budgets.

I was asking more in terms of Gb/s rather than MHz of bandwidth. Dedicated 
ground stations with bigger antennas, better filters, more processing and 
overall a much higher budget can get much better data rates out of a given 
amount of bandwidth than the user end stations will.

it's also possible (especially with bigger antennas) for one ground station 
location to talk to multiple different satellites at once (the aiming of the 
antennas can isolate the signals from each other)

>> As latency changes, figuring out if it's extra distance that must be 
>> traveled, or buffering is hard. does the latency stay roughly the same until 
>> the next satellite change? or does it taper off?

> Good question. You would expect step changes in physical latency between 
> satellites, but also gradual change related to satellite movement. Plus of 
> course any rubble thrown into any queue by something suddenly turning up on 
> that path. Don't forget that it's not just cells now, we're also talking up- 
> and downlink for the laser ISLs, at least in some places.

how far do the satellites move in 15 min and what effect would that have on 
latency (I would assume that most of the time, the satellites are switched to as 
they are getting nearer the two stations, so most of the time, I would expect a 
slight reduction in latency for ~7 min and then a slight increase for ~7 min, 
but I would not expect that this would be a large variation

>> If it stays the same, I would suspect that you are actually hitting a 
>> different ground station and there is a VPN backhaul to your egress point to 
>> the regular Internet (which doesn't support mobile IP addresses) for that 
>> cycle. If it tapers off, then I could buy bufferbloat that gets resolved as 
>> TCP backs off.
>
> Yes, quite sorting out which part of your latency is what is the million 
> dollar question here...
>
> We saw significant RTT changes here during the recent cyclone over periods of 
> several hours, and these came in steps (see below), with the initial change 
> being a downward one. Averages are over 60 pings (the time scale isn't 100% 
> true as we used "one ping, one second" timing) here.
>
>
> We're still not sure whether to attribute this to load change or ground 
> station changes. There were a lot of power outages, especially in Auckland's 
> lifestyle block belt, which teems with Starlink users, but all three North 
> Island ground stations were also in areas affected by power outages (although 
> the power companies concerned don't provide the level of detail to establish 
> whether they were affected). It's also not clear what, if any, backup power 
> arrangements they have). At ~25 ms, the step changes in RTT are too large be 
> the result of a switch in ground stations, though, the path differences just 
> aren't that large. You'd also expect a ground station outage to result in 
> longer RTTs, not shorter ones, if you need to re-route via another ground 
> station. One explanation might be users getting cut off if they relied on one 
> particular ground station for bent pipe ops - but that would not explain this 
> order of magnitude effect as I'd expect that number to be small. So maybe 
> power outages at the user end after all. But that would then tell us that 
> these are load-dependent queuing delays. Moreover, since those load changes 
> wouldn't have involved the router at our site, we can conclude that these are 
> queue sojourn times in the Starlink network.

I have two starlink dishes in the southern california area, I'm going to put 
one on the low-priority mobile plan shortly. These are primarily used for backup 
communication, so I would be happy to add something to them to do latency 
monitoring. In looking at what geo-location reports my location as, I see it 
wander up and down the west coast, from the Los Angeles area all the way up to 
Canada.

>> I think that active queue management on the sending side of the bottleneck 
>> will handle it fairly well. It doesn't have to do calculations based on what 
>> the bandwidth is, it just needs to know what it has pending to go out.

> Understood - but your customer for AQM is the sending TCP client, and there 
> are two questions here: (a) Does your AQM handle rapid load changes and (b) 
> how do your TCP clients actually respond to your AQM's handling?

AQM allocates the available bandwidth between different connections (usually 
different users)

When it does this indirectly for inbound traffic by delaying acks, the results 
depend on the senders handling of these indirect signals that were never 
intended for this purpose.

But when it does this directly on the sending side, it doesn't matter what the 
senders want, their data WILL be managed to the priority/bandwidth that the AQM 
sets, and eventually their feedback is dropped packets, which everyone who is 
legitimate responds to. But even if they don't respond (say a ping flood or DoS 
attack), the AQM will limit the damage to that connection, allowing the other 
connections trying to use that link to continue to function.

David Lang