[Starlink] Starlink in Kiribati

[Starlink] Starlink in Kiribati

[Ulrich Speidel]

Some interesting insight yesterday: One of my PhD student Wayne Reiher's 
contacts on Tarawa atoll in Kiribati (pronounced "Kiri-bus"), Karotu 
Tannang, operates a Starlink kit there, around 4,200 km north of 
Auckland, New Zealand. Wayne asked him to run a few traceroutes and five 
minute pings with large packets to get us an initial idea as to how well 
it worked and how the signal got back to the rest of the world.

Tarawa is well out of bent-pipe range of any known Starlink gateways, so 
would have to rely on laser ISLs.

All traces (to NZ, Chile, the US, Germany and Japan) exited the SpaceX 
address space in New Zealand to a variety of upstream providers. IP 
addresses on the Starlink side of the trace beyond Dishy are, in order 
of appearance in the traces: 100.64.0.1, 172.16.249.6, followed by 
149.19.109.30 if exiting to Hurricane Electric as upstream or 
149.19.109.34 if exiting to Kinect. All those addresses have RTT 
indicating that they are in NZ.

Lowest RTT seen to the first NZ hop was 65 ms, with values between 100 
and 200 ms being the most common.

RTTs to the US were mostly in the high 200 ms.

Packet loss was atrocious: about 5% to NZ, 8% to Japan and Chile, 9% to 
the US. Hard to say whether this was just due to variation on the leg to 
NZ or whether anything got dropped further down the road.

First thoughts:

1) There is no 53 / 53.2 deg orbital plane for Starlink that is 
simultaneously visible from both Tarawa and NZ. Unless the traceroutes 
all happened to be piggybacking on one of the few polar orbit passing 
overhead, this implies at least some degree of cross-plane forwarding. 
We'll ask Karotu to run a few more tests for us at different times.

2) The fact that we don't see more than the "usual" number of Starlink 
IP addresses in the tracreroutes indicates that whatever IP routing may 
be happening on satellites that handle the traffic via ISLs happens at a 
tunnel layer further down the stack.

3) The fact that the traffic emerges in New Zealand regardless of global 
destination also indicates that the Starlink network uses a tunnel based 
on Dishy location and a nearby gateway but does not attempt to route to 
final destination at this point in time.

The 65 ms RTT also tells us a few things. For one, at 4,200 km great 
circle distance on the ground, the dishy-to-gateway physical path would 
be at least 5,000 km even if all lined up with a polar orbital plane 
involved. That makes 10,000 km of RTT path, which translates into about 
33 ms of propagation RTT. If cross-plane routing were involved here, 
we'd get a zig-zag path - so roughly 1 1/2 times longer. Makes about 50 
ms. In-plane only routing would involve a gateway in Australia (similar 
length physical path dishy to gateway) along with a 2,000 km 
trans-Tasman cable leg. The 2000 km cable leg would be equivalent to 
about 20 ms of additional RTT over the 10,000 km space RTT path, so that 
could in principle also work. Quite why everything would emerge in 
Auckland though in this case would be a mystery to me.



-- 
****************************************************************
Dr. Ulrich Speidel

School of Computer Science

Room 303S.594 (City Campus)

The University of Auckland
u.speidel@auckland.ac.nz
http://www.cs.auckland.ac.nz/~ulrich/
****************************************************************

Re: [Starlink] Starlink in Kiribati

[Michael Richardson]

[-- Attachment #1: Type: text/plain, Size: 2802 bytes --]


Ulrich Speidel via Starlink <starlink@lists.bufferbloat.net> wrote:
    > All traces (to NZ, Chile, the US, Germany and Japan) exited the SpaceX
    > address space in New Zealand to a variety of upstream providers. IP
    > addresses on the Starlink side of the trace beyond Dishy are, in order of
    > appearance in the traces: 100.64.0.1, 172.16.249.6, followed by 149.19.109.30
    > if exiting to Hurricane Electric as upstream or 149.19.109.34 if exiting to
    > Kinect. All those addresses have RTT indicating that they are in NZ.

What are you being NAT44'ed to?
Presumably, that IP is in NZ, and you could traceroute/ping back to it from
elsewhere to be able to subject whatever congestion is beyond the peering point.

    > Lowest RTT seen to the first NZ hop was 65 ms, with values between 100 and
    > 200 ms being the most common.

    > RTTs to the US were mostly in the high 200 ms.

    > 2) The fact that we don't see more than the "usual" number of Starlink IP
    > addresses in the tracreroutes indicates that whatever IP routing may be
    > happening on satellites that handle the traffic via ISLs happens at a tunnel
    > layer further down the stack.

Given Musk's age old tweet that it was "simpler than IPv6", and that we know
that it's some kind of broadcom SDN chipset, this makes sense.
I wish they had used SR6, and done IPv4 as a service.

    > 3) The fact that the traffic emerges in New Zealand regardless of global
    > destination also indicates that the Starlink network uses a tunnel based
    > on Dishy location and a nearby gateway but does not attempt to route to final
    > destination at this point in time.

I'm not surprised about this.
I don't imagine they can world-wide stuff until all the non-ISL birds have aged out.

    > The 65 ms RTT also tells us a few things. For one, at 4,200 km great circle
    > distance on the ground, the dishy-to-gateway physical path would be at least
    > 5,000 km even if all lined up with a polar orbital plane involved. That makes
    > 10,000 km of RTT path, which translates into about 33 ms of propagation
    > RTT. If cross-plane routing were involved here, we'd get a zig-zag path - so
    > roughly 1 1/2 times longer. Makes about 50 ms. In-plane only routing would
    > involve a gateway in Australia (similar length physical path dishy to
    > gateway) along with a 2,000 km trans-Tasman cable leg. The 2000 km cable leg
    > would be equivalent to about 20 ms of additional RTT over the 10,000 km space
    > RTT path, so that could in principle also work. Quite why everything would
    > emerge in Auckland though in this case would be a mystery to me.

I think you are saying that the your lowest RTT of 65ms is easily supported
by physical distances alone?
And that it can't get better.




[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 511 bytes --]

Re: [Starlink] Starlink in Kiribati

[Ulrich Speidel]

[-- Attachment #1: Type: text/plain, Size: 5399 bytes --]

On 22/04/2023 4:11 am, Michael Richardson wrote:
>
> Ulrich Speidel via Starlink <starlink@lists.bufferbloat.net> wrote:
> > All traces (to NZ, Chile, the US, Germany and Japan) exited the SpaceX
> > address space in New Zealand to a variety of upstream providers. IP
> > addresses on the Starlink side of the trace beyond Dishy are, in 
> order of
> > appearance in the traces: 100.64.0.1 
> <http://100.64.0.1>, 
> 172.16.249.6, followed by 149.19.109.30 
> <http://149.19.109.30>
> > if exiting to Hurricane Electric as upstream or 149.19.109.34 
> <http://149.19.109.34> 
> if exiting to
> > Kinect. All those addresses have RTT indicating that they are in NZ.
>
> What are you being NAT44'ed to?
Good question - can't answer this right now as Dishy is back in its 
suitcase in our lab ;-)
> Presumably, that IP is in NZ, and you could traceroute/ping back to it 
> from
> elsewhere to be able to subject whatever congestion is beyond the 
> peering point.
Beyond peering point is an interesting thing, too. We spent a day this 
week on a mountain 5 hours drive south of Auckland running Dishy out of 
the back of my car, just to find that the one traceroute we took from 
there shows a routing issue on the Internet side of the POP - HE sent 
traffic from NZ to Australia, from where it was returned to us, adding 
over 4000 km of cable to the path each way. Sigh. Ookla tests (to a 
different machine in a different network in NZ) gave around half the 
download rates we were observing from the two other locations we visited 
(and we took 60-180 samples in each location), where that routing issue 
didn't show up.
>
> > Lowest RTT seen to the first NZ hop was 65 ms, with values between 
> 100 and
> > 200 ms being the most common.
>
> > RTTs to the US were mostly in the high 200 ms.
>
> > 2) The fact that we don't see more than the "usual" number of 
> Starlink IP
> > addresses in the tracreroutes indicates that whatever IP routing may be
> > happening on satellites that handle the traffic via ISLs happens at 
> a tunnel
> > layer further down the stack.
>
> Given Musk's age old tweet that it was "simpler than IPv6", and that 
> we know
> that it's some kind of broadcom SDN chipset, this makes sense.
> I wish they had used SR6, and done IPv4 as a service.
Indeed, but then who would want to rely on something he said a while ago?
>
> > 3) The fact that the traffic emerges in New Zealand regardless of global
> > destination also indicates that the Starlink network uses a tunnel based
> > on Dishy location and a nearby gateway but does not attempt to route 
> to final
> > destination at this point in time.
>
> I'm not surprised about this.
> I don't imagine they can world-wide stuff until all the non-ISL birds 
> have aged out.
Well, they do have a complete constellation with lasers, and a polar one 
that's semi-complete, so in theory they should be able to do this...
>
> > The 65 ms RTT also tells us a few things. For one, at 4,200 km great 
> circle
> > distance on the ground, the dishy-to-gateway physical path would be 
> at least
> > 5,000 km even if all lined up with a polar orbital plane involved. 
> That makes
> > 10,000 km of RTT path, which translates into about 33 ms of propagation
> > RTT. If cross-plane routing were involved here, we'd get a zig-zag 
> path - so
> > roughly 1 1/2 times longer. Makes about 50 ms. In-plane only routing 
> would
> > involve a gateway in Australia (similar length physical path dishy to
> > gateway) along with a 2,000 km trans-Tasman cable leg. The 2000 km 
> cable leg
> > would be equivalent to about 20 ms of additional RTT over the 10,000 
> km space
> > RTT path, so that could in principle also work. Quite why everything 
> would
> > emerge in Auckland though in this case would be a mystery to me.
>
> I think you are saying that the your lowest RTT of 65ms is easily 
> supported
> by physical distances alone?
> And that it can't get better.

Basically, an observed RTT puts an upper bound on the length of the 
possible physical path a packet can have taken. On that basis, I'm 
simply looking at potentially feasible paths here. I'm just observing 
that if you are landing traffic in Australia where you have multiple 
gateways and POPs, it makes little sense to then also have a terrestrial 
tunnel to NZ for the traffic that you are landing and that's destined to 
other continents.

I mean, I'd like Auckland to be the navel of the world, but I somehow 
doubt that Elon would take the same view ... Then again, at least one of 
his Silicon Valley mates got NZ citizenship after just two weeks in the 
country. The guy was going to promote this widely but then somehow that 
didn't happen and if I correctly remember, it took an Official 
Information Act request to find out that the guy had become a proud 
Kiwi. FAIK he hasn't spent much if any time here since becoming a 
citizen either. So who knows whether Elon might have a secret bolt hole 
here? Lots of people do...


-- 
****************************************************************
Dr. Ulrich Speidel

School of Computer Science

Room 303S.594 (City Campus)

The University of Auckland
u.speidel@auckland.ac.nz  
http://www.cs.auckland.ac.nz/~ulrich/
****************************************************************



[-- Attachment #2: Type: text/html, Size: 7246 bytes --]