[Starlink] Starlink hidden buffers

[Ulrich Speidel]

On 25/05/2023 1:59 am, David Lang wrote:
>
> >> >>
> >> 
> https://www.amazon.com/YAOSHENG-Rectangular-Adapter-Connect-Injector/dp/B0BYJTHX4P 
> <https://www.amazon.com/YAOSHENG-Rectangular-Adapter-Connect-Injector/dp/B0BYJTHX4P> 
>
> >> 
> <https://www.amazon.com/YAOSHENG-Rectangular-Adapter-Connect-Injector/dp/B0BYJTHX4P 
> <https://www.amazon.com/YAOSHENG-Rectangular-Adapter-Connect-Injector/dp/B0BYJTHX4P>>
> >>
> >> >> >>
> >> >> > I'll see whether I can get hold of one of these. Cutting a 
> cable on a
> >> >> > university IT asset as an academic is not allowed here, except 
> if it
> >> >> > doesn't meet electrical safety standards.
> > OK, we have one on order, along with PoE injector and power supply. 
> Don't
> > hold your breath, though, I'll be out of the country when it arrives 
> and
> > it'll be late July before I get to play with it.
>
> I've got a couple on order, but they won't arrive for 1-3 more weeks :-(
I envy you!
> I'll also note that in the last launch of the v2 mini satellites, they 
> mentioned
> that those now supported E band backhaul to handle 4x the bandwidth of 
> the
> earlier satellites
Still not enough to connect the missing 2.5 or so billion, but a step in 
the right direction for sure.
>
> > It's certainly noticeable here that they seem to have sets of three 
> grouped
> > together in a relatively compact geographical area (you could visit 
> all NZ
> > North Island ground stations in a day by car from Auckland, Auckland 
> traffic
> > notwithstanding, and at a stretch could do the same down south from 
> Hinds to
> > Awarua if you manage to ignore the scenery, but getting from the 
> southernmost
> > North Island ground station to the northernmost South Island one is 
> basically
> > a two day drive plus ferry trip).
>
> I lived in Wanganui for a few years, including one RV trip down the South
> Island. I know what you mean about needing to ignore the scenery :-)
Interesting - that must have been before the local īwi pointed out once 
again that the town had misspelled its name since 1854, and for once 
were heard - so it's now officially "Whanganui", for crown agencies, anyway.
> Ok, I thought I had heard they switched every 15 min, so it's every 5 min
> instead?
Dishy collects this information as a cumulative dataset, which the tools 
query via grpc. The frames in the movie corresponds to snapshots of the 
dataset taken at 5 second intervals. This indicates switches roughly 
every ten to seventy seconds, with most dwell times being around 15-30 
seconds.
>
> > Conclusion: latency change from tracking one satellite is smaller 
> than the
> > latency difference as you jump between satellites. You could be 
> looking at
> > several 100 km of path difference here. In an instant. Even that, at 
> 300,000
> > km/s of propagation speed, is only in the order of maybe 1 ms or so 
> - peanuts
> > compared to the RTTs in the dozens of ms that we're seeing. But if 
> you get
> > thrown from one queue onto another as you get handed over - what 
> does that do
> > to the remote TCP stack that's serving you?
>
> yes, the point I thought that I was trying to make was that the 
> latency change
> from satellite movement was not very significant
So it's got to come from somewhere else.
>
> >> >> If it stays the same, I would suspect that you are actually 
> hitting a
> >> >> different ground station and there is a VPN backhaul to your 
> egress point
> >> >> to the regular Internet (which doesn't support mobile IP 
> addresses) for
> >> >> that cycle. If it tapers off, then I could buy bufferbloat that 
> gets
> >> >> resolved as TCP backs off.
> >> >
> >> > Yes, quite sorting out which part of your latency is what is the 
> million
> >> > dollar question here...
> >> >
> >> > We saw significant RTT changes here during the recent cyclone 
> over periods
> >> > of several hours, and these came in steps (see below), with the 
> initial
> >> > change being a downward one. Averages are over 60 pings (the time 
> scale
> >> > isn't 100% true as we used "one ping, one second" timing) here.
> >> >
> >> >
> >> > We're still not sure whether to attribute this to load change or 
> ground
> >> > station changes. There were a lot of power outages, especially in
> >> > Auckland's lifestyle block belt, which teems with Starlink users, 
> but all
> >> > three North Island ground stations were also in areas affected by 
> power
> >> > outages (although the power companies concerned don't provide the 
> level of
> >> > detail to establish whether they were affected). It's also not 
> clear what,
> >> > if any, backup power arrangements they have). At ~25 ms, the step 
> changes
> >> > in RTT are too large be the result of a switch in ground 
> stations, though,
> >> > the path differences just aren't that large. You'd also expect a 
> ground
> >> > station outage to result in longer RTTs, not shorter ones, if you 
> need to
> >> > re-route via another ground station. One explanation might be 
> users getting
> >> > cut off if they relied on one particular ground station for bent 
> pipe ops -
> >> > but that would not explain this order of magnitude effect as I'd 
> expect
> >> > that number to be small. So maybe power outages at the user end 
> after all.
> >> > But that would then tell us that these are load-dependent queuing 
> delays.
> >> > Moreover, since those load changes wouldn't have involved the 
> router at our
> >> > site, we can conclude that these are queue sojourn times in the 
> Starlink
> >> > network.
>
> remember that SpaceX controlls the ground stations as well, so if they 
> are doing
> any mobile IP trickery to redirect traffic from one ground station to 
> another,
> they can anticipate the shift or move the queue for the user or other 
> trickery
> like this (probably aren't yet, they seem to be in the early days 
> here, focusing
> on keeping things working and improving on the space side more than 
> anything
> else)
I strongly suspect that they are experimenting with this here and with 
that there.
>
>
> >> AQM allocates the available bandwidth between different connections 
> (usually
> >> different users)
> > But it does this under the assumption that the vector for changes in 
> bandwidth
> > availability is the incoming traffic, which AQM gives (indirect) 
> feedback to,
> > right?
>
> no, this is what I'm getting at below
>
> >> When it does this indirectly for inbound traffic by delaying acks, the
> >> results depend on the senders handling of these indirect signals 
> that were
> >> never intended for this purpose.
>
> This is what you are thinking of, where it's providing indirect 
> feedback to an
> unknowable inbound queue on a remote system
>
> >> But when it does this directly on the sending side, it doesn't 
> matter what
> >> the senders want, their data WILL be managed to the 
> priority/bandwidth that
> >> the AQM sets, and eventually their feedback is dropped packets, which
> >> everyone who is legitimate responds to.
>
> when the AQM in on the sending side of the bottleneck, it now has 
> direct control
> over the queue, and potentially has information over the available 
> bandwidth as
> it changes. But even if it doesn't know what the available bandwidth 
> is, it
> still can dispatch the data in it's queues 'fairly' (whatever that 
> means to the
> particulat AQM algorithm), changes in the data rate just change how 
> fast the
> queue drains.

Yes - but if you delay ACKs, the only entity this has any effect on is 
the original (remote) TCP sender, which is who you are trying to 
persuade to take it easy so you're not going to be forced to (tail or 
otherwise) drop packets.

Dropping helps clear your queue (the one in front of the bottleneck).

>
> > Understood. You build a control loop, where the latency is the delay 
> in the
> > control signal.
> >
> > Classically, you have a physical bottleneck that the AQM manages, 
> where the
> > physical bandwidth doesn't change.
> >
> > The available bandwidth changes, (mostly) as a result of TCP 
> connections (or
> > similarly behaved UDP applications) joining in slow start, or 
> disappearing.
> >
> > Basically, your queues grow and shrink one packet at a time.
> >
> > Your control signal allows you (if they're well behaved) throttle /
> > accelerate senders.
> >
> > What you don't get are quantum jumps in queue occupancy, jump 
> changes in
> > underlying physical bandwidth, or a whole set of new senders that are
> > completely oblivious to any of your previous control signals. But 
> you get all
> > that with satellite handovers like these.
>
> for a single TCP session,it has slow-start, but if you suddently start 
> dozens or
> hundreds of TCP sessions, (bittorrent, other file transfer protocols, 
> or just a
> website with hundreds of sub-elements), I think it's a bigger step 
> than you are
> thinking.
Doesn't each TCP session maintain and manage its own cwnd?
>
> And again, I think the same issue exists on cell sites as users move 
> from one
> cell to another.
Yes. But that happens gradually in comparison to Starlink, and the only 
TCP stack that potentially gets affected badly as a user moves from one 
cell site to the next is that of the user. But what you have here is the 
equivalent of the cell tower moving out of range of a whole group of 
users in one go. Different ballpark?
>
> > So what if the response you elicit in this way is to a queue 
> scenario that no
> > longer applies?
>
> you run the risk of under-utilizing the link for a short time (which 
> may mean
> that you decide to run the queues a little bigger than with fixed 
> links, so that
> when a chunk of data disappears from your queue, you still will keep 
> utilization
> up, sacraficing some latency to improve overall throughput)
So we're back to the "more buffer" scenario here, too.
>
> David Lang 

-- 
****************************************************************
Dr. Ulrich Speidel

School of Computer Science

Room 303S.594 (City Campus)

The University of Auckland
u.speidel at auckland.ac.nz  
http://www.cs.auckland.ac.nz/~ulrich/
****************************************************************


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.bufferbloat.net/pipermail/starlink/attachments/20230525/ffb7b787/attachment-0001.html>