On Mon, 15 May 2023, Ulrich Speidel wrote:

> On 14/05/2023 9:00 pm, David Lang wrote:
>> On Sun, 14 May 2023, Ulrich Speidel wrote:
>>
>> >> I just discovered that someone is manufacturing an adapter so you no
>> >> longer have to cut the cable
>> >>
>> >>
>> https://www.amazon.com/YAOSHENG-Rectangular-Adapter-Connect-Injector/dp/B0BYJTHX4P

>> >>
>> > I'll see whether I can get hold of one of these. Cutting a cable on a
>> > university IT asset as an academic is not allowed here, except if it
>> > doesn't meet electrical safety standards.

>> >
>> > Alternatively, has anyone tried the standard Starlink Ethernet adapter with
>> > a PoE injector instead of the WiFi box? The adapter above seems to be like
>> > the Starlink one (which also inserts into the cable between Dishy and
>> > router).
>>
>> that connects you a 2nd ethernet port on the router, not on the dishy
>>
>> I just ordered one of those adapters, it will take a few weeks to arrive.
> How do we know that the Amazon version doesn't do the same?

because it doesn't involve the router at all. It allows you to replace the
router with anything you want.

People have documented how to cut the cable and crimp on a RJ45 connector, use a
standard PoE injector, and connect to any router you want. I was preparing to do
that (and probably still will for one cable to use a different locations to
avoid having a 75 ft cable from the dish mounted on the roof of my van to the
router a couple feet away), This appears to allow me to do the same functional
thing, but without cutting the cable.

>> >> > I suspect they're handing over whole cells, not individual users, at a
>> >> > time.
>> >>
>> >> I would guess the same (remember, in spite of them having launched >4000
>> >> satellites, this is still the early days, with the network changing as
>> >> more launching)
>> >>
>> >> We've seen that it seems that there is only one satellite serving any cell
>> >> one time.
>>
>> > But the reverse is almost certainly not true: Each satellite must serve
>> > multiple cells.
>>
>> true, but while the satellite over a given area will change, the usage in
>> that area isn't changing that much

> Exactly. But your underlying queue sits on the satellite, not in the area.

only if the satellite is where you have more input than output. That may be the
case for users uploading, but for users downloading, I would expect that the
bandwidth bottleneck would be from the Internet connected ground station to the
satellite, with the satellite serving many cells but only having one uplink.

Leaving lasers aside for the moment.

I'd expect there to be one queue for each satellite uplink at the gateway ground station, and that the occupancy of that queue depends on how much demand the users on that satellite currently produce. So as a remote terminal switches satellites, even if the ground station remains the same, it sees different queuing delays for its inbound traffic at the ground station.

For the uplink from the user terminal, we can't have multiple users accessing the same uplink channel (however you define "channel" - frequency, time slot, spreading code, beam, polarisation, any combination thereof, ...) simultaneously as they are not able to coordinate and you wouldn't want random access for your main data link channel because of the hidden node collisions this would produce (a random access channel paired with an access grant channel is a different story). So you'd get slot assignments from the satellite obviously, and the queue for one of these sits at the user terminal. But what isn't clear to me is whether the satellites are truly only handled by a single ground station at a time, or perhaps by multiple ground stations. If it's the latter, then you might end up with a situation where you have more traffic arriving at the satellite than it can dispatch to its ground station(s), and then you'd need a queue in the uplink direction also.

Similarly, if the combined uplinks from the ground station are able to deliver more data than the satellite can downlink to its users through its current slot assignments, we need a queuing system on the satellite in that direction, too.

Add lasers in, and it seems like having some sort of buffer on the satellites is a must.  

>> >> But remember that the system does know how much usage there is in the cell
>> >> before they do the handoff. It's unknown if they do anything with that, or
>> >> if they are just relaying based on geography. We also don't know what the
>> >> bandwidth to the ground stations is compared to the dishy.
>>
>> > Well, we do know for NZ, sort of, based on the licences Starlink has here.
>>
>> what is the ground station bandwith?
>
> https://rrf.rsm.govt.nz/ui/search/licence - seach for "Starlink"
>
> ...all NZ licences in all their glory. Looking at Starlink SES (satellite
> earth station) TX (which is the interesting direction I guess):
>
> - Awarua, Puwera, Hinds, Clevedon, Cromwell, Te Hana: 29750.000000 TX (BW =
> 500 MHz)
> - Awarua, Puwera, Hinds, Clevedon, Cromwell, Te Hana: 28850.000000 TX (BW =
> 500 MHz)
> - Awarua, Puwera, Hinds, Clevedon, Cromwell, Te Hana: 28350.000000 TX (BW =
> 500 MHz)
> - Awarua, Puwera, Hinds, Clevedon, Cromwell, Te Hana: 28250.000000 TX (BW =
> 500 MHz)
> - Awarua, Puwera, Hinds, Clevedon, Cromwell, Te Hana: 27750.000000 TX (BW =
> 500 MHz)
>
> So 2.5 GHz up, licensed from 6 ground stations. Now I'm not convinced that
> they would use all of those from all locations simultaneously because of the
> risk of off-beam interference. They'll all be transmitting south, ballpark.
> If there was full re-use at all ground stations, we'd be looking at 15 GHz.
> If they are able to re-use on all antennas at each ground station, then we're
> looking at 9 golf balls each in Puwera, Te Hana, Clevedon, Hinds and
> Cromwell, and an unknown number at Awarua. Assuming 9 there, we'd be looking
> at 135 GHz all up max.
>
> Awarua and Cromwell are 175 km apart, Hinds another 220 km from Cromwell,
> then it's a hop of about 830 km to Clevedon, and from there another 100 km to
> Te Hana, which is another 53 km from Puwera, so keeping them all out of each
> other's hair all the time might be a bit difficult.
>
> Lots of other interesting info in the licenses, such as EIRP, in case you're
> wanting to do link budgets.

I was asking more in terms of Gb/s rather than MHz of bandwidth. Dedicated
ground stations with bigger antennas, better filters, more processing and
overall a much higher budget can get much better data rates out of a given
amount of bandwidth than the user end stations will.

it's also possible (especially with bigger antennas) for one ground station
location to talk to multiple different satellites at once (the aiming of the
antennas can isolate the signals from each other)

Well, the Gb/s is what a link budget would give you if you knew the modulation scheme(s) and any FEC used. The ground station antennas are normal parabolic dishes in radomes for all we can tell, and are all the same size, so you can kind of estimate aperture and hence gain reasonably well. Path loss depends a little on distance and path quality (weather / rain fade), and we don't really know in how far their modems use adaptive rates to cope with this (my Ookla tests during Cyclone Gabrielle most certainly don't rule this out - rates went down both ways during the thick of it). I guess we know relatively little about the on-board phased array on the satellites (apart from very loose bounds), which restricts our ability to say much about gain there (and potential for spatial separation / re-use of beam frequencies). We also don't know how Starlink manages its frequency allocation across its ground stations.

It's certainly noticeable here that they seem to have sets of three grouped together in a relatively compact geographical area (you could visit all NZ North Island ground stations in a day by car from Auckland, Auckland traffic notwithstanding, and at a stretch could do the same down south from Hinds to Awarua if you manage to ignore the scenery, but getting from the southernmost North Island ground station to the northernmost South Island one is basically a two day drive plus ferry trip).

>> As latency changes, figuring out if it's extra distance that must be
>> traveled, or buffering is hard. does the latency stay roughly the same until
>> the next satellite change? or does it taper off?

> Good question. You would expect step changes in physical latency between
> satellites, but also gradual change related to satellite movement. Plus of
> course any rubble thrown into any queue by something suddenly turning up on
> that path. Don't forget that it's not just cells now, we're also talking up-
> and downlink for the laser ISLs, at least in some places.

how far do the satellites move in 15 min and what effect would that have on
latency (I would assume that most of the time, the satellites are switched to as
they are getting nearer the two stations, so most of the time, I would expect a
slight reduction in latency for ~7 min and then a slight increase for ~7 min,
but I would not expect that this would be a large variation

Dishy tracks most satellites for significantly less than 15 minutes, and for a relatively small part of their orbit. Let me explain:

This is an obstruction map obtained with starlink-grpc-tools (https://github.com/sparky8512/starlink-grpc-tools). The way to read this is in polar coordinates: The centre of the image is the dishy boresight (direction of surface normal), distance from the centre is elevation measured as an angle from the surface normal, and direction from the centre is essentially the azimuth - top is north, left is west, bottom is south, and right is east. The white tracks are the satellites dishy uses, and a graph like this gets built up over time, one track at a time. Notice how short the tracks are - they don't follow the satellite for long - typically under a minute. The red bits are satellites getting obscured by the edge of our roof.

I've also attached a time lapse movie of how one of these graphs builds up - if I correctly remember (the script is on another machine), one frame in the video corresponds to 5 seconds.

Conclusion: latency change from tracking one satellite is smaller than the latency difference as you jump between satellites. You could be looking at several 100 km of path difference here. In an instant. Even that, at 300,000 km/s of propagation speed, is only in the order of maybe 1 ms or so - peanuts compared to the RTTs in the dozens of ms that we're seeing. But if you get thrown from one queue onto another as you get handed over - what does that do to the remote TCP stack that's serving you? 

>> If it stays the same, I would suspect that you are actually hitting a
>> different ground station and there is a VPN backhaul to your egress point to
>> the regular Internet (which doesn't support mobile IP addresses) for that
>> cycle. If it tapers off, then I could buy bufferbloat that gets resolved as
>> TCP backs off.
>
> Yes, quite sorting out which part of your latency is what is the million
> dollar question here...
>
> We saw significant RTT changes here during the recent cyclone over periods of
> several hours, and these came in steps (see below), with the initial change
> being a downward one. Averages are over 60 pings (the time scale isn't 100%
> true as we used "one ping, one second" timing) here.
>
>
> We're still not sure whether to attribute this to load change or ground
> station changes. There were a lot of power outages, especially in Auckland's
> lifestyle block belt, which teems with Starlink users, but all three North
> Island ground stations were also in areas affected by power outages (although
> the power companies concerned don't provide the level of detail to establish
> whether they were affected). It's also not clear what, if any, backup power
> arrangements they have). At ~25 ms, the step changes in RTT are too large be
> the result of a switch in ground stations, though, the path differences just
> aren't that large. You'd also expect a ground station outage to result in
> longer RTTs, not shorter ones, if you need to re-route via another ground
> station. One explanation might be users getting cut off if they relied on one
> particular ground station for bent pipe ops - but that would not explain this
> order of magnitude effect as I'd expect that number to be small. So maybe
> power outages at the user end after all. But that would then tell us that
> these are load-dependent queuing delays. Moreover, since those load changes
> wouldn't have involved the router at our site, we can conclude that these are
> queue sojourn times in the Starlink network.

I have two starlink dishes in the southern california area, I'm going to put
one on the low-priority mobile plan shortly. These are primarily used for backup
communication, so I would be happy to add something to them to do latency
monitoring. In looking at what geo-location reports my location as, I see it
wander up and down the west coast, from the Los Angeles area all the way up to
Canada.

>> I think that active queue management on the sending side of the bottleneck
>> will handle it fairly well. It doesn't have to do calculations based on what
>> the bandwidth is, it just needs to know what it has pending to go out.

> Understood - but your customer for AQM is the sending TCP client, and there
> are two questions here: (a) Does your AQM handle rapid load changes and (b)
> how do your TCP clients actually respond to your AQM's handling?

AQM allocates the available bandwidth between different connections (usually
different users)

When it does this indirectly for inbound traffic by delaying acks, the results
depend on the senders handling of these indirect signals that were never
intended for this purpose.

But when it does this directly on the sending side, it doesn't matter what the
senders want, their data WILL be managed to the priority/bandwidth that the AQM
sets, and eventually their feedback is dropped packets, which everyone who is
legitimate responds to.

Understood. You build a control loop, where the latency is the delay in the control signal.

Classically, you have a physical bottleneck that the AQM manages, where the physical bandwidth doesn't change.

The available bandwidth changes, (mostly) as a result of TCP connections (or similarly behaved UDP applications) joining in slow start, or disappearing.

Basically, your queues grow and shrink one packet at a time.

Your control signal allows you (if they're well behaved) throttle / accelerate senders.

What you don't get are quantum jumps in queue occupancy, jump changes in underlying physical bandwidth, or a whole set of new senders that are completely oblivious to any of your previous control signals. But you get all that with satellite handovers like these.

So what if the response you elicit in this way is to a queue scenario that no longer applies?

But even if they don't respond (say a ping flood or DoS
attack), the AQM will limit the damage to that connection, allowing the other
connections trying to use that link to continue to function.

David Lang

-- 
****************************************************************
Dr. Ulrich Speidel

School of Computer Science

Room 303S.594 (City Campus)

The University of Auckland
u.speidel@auckland.ac.nz 
http://www.cs.auckland.ac.nz/~ulrich/
****************************************************************