From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from au-smtp-delivery-117.mimecast.com (au-smtp-delivery-117.mimecast.com [103.96.23.117]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id EF08D3CB37 for ; Fri, 21 Apr 2023 19:04:21 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=auckland.ac.nz; s=mimecast20200506; t=1682118259; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=hl0LuGkUONwpeDoTJezYCT6WGcYj2PBBQq1fO37UblA=; b=WjUB4lDSvn1jpxIAkebfvYmri0wSR4yRnZc+49Wz5cOQDDgO9qP3HJx4nYc9AKd+k3TBk4 QZ37lYdqTiaOgOe4RMCQZTwj0+oCfr9pOqTcn5DxUinf5A6eJ9cpuD8+msT4qpn9EVxwWF pxbT4T2xgva8nS4ZDnLaiAOVCnF7W/0= Received: from AUS01-SY4-obe.outbound.protection.outlook.com (mail-sy4aus01lp2177.outbound.protection.outlook.com [104.47.71.177]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id au-mta-62-fcOxl1JaMsicrRwynpTY-Q-1; Sat, 22 Apr 2023 09:04:17 +1000 X-MC-Unique: fcOxl1JaMsicrRwynpTY-Q-1 Received: from SY4PR01MB6979.ausprd01.prod.outlook.com (2603:10c6:10:142::13) by SY4PR01MB6571.ausprd01.prod.outlook.com (2603:10c6:10:124::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6319.22; Fri, 21 Apr 2023 23:04:15 +0000 Received: from SY4PR01MB6979.ausprd01.prod.outlook.com ([fe80::68d5:4e6b:745e:197e]) by SY4PR01MB6979.ausprd01.prod.outlook.com ([fe80::68d5:4e6b:745e:197e%6]) with mapi id 15.20.6319.022; Fri, 21 Apr 2023 23:04:15 +0000 Message-ID: <776fcc5a-06db-5b7e-825c-a9fe9c1ad6d2@auckland.ac.nz> Date: Sat, 22 Apr 2023 11:04:13 +1200 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.10.0 To: Michael Richardson , "starlink@lists.bufferbloat.net" References: <28149.1682093487@localhost> From: Ulrich Speidel In-Reply-To: <28149.1682093487@localhost> X-ClientProxiedBy: SYAPR01CA0045.ausprd01.prod.outlook.com (2603:10c6:1:1::33) To SY4PR01MB6979.ausprd01.prod.outlook.com (2603:10c6:10:142::13) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SY4PR01MB6979:EE_|SY4PR01MB6571:EE_ X-MS-Office365-Filtering-Correlation-Id: a7683534-dc28-4a5c-1662-08db42bcba90 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0 X-Microsoft-Antispam-Message-Info: 6+CNh7myIPcBdKDyt8dxNmtU5OVni0kgess5017w7WTBdTegJB65Lr+Ahsf+Aae6/fn77CBe/BZPjSzM8vxKQbcC/iMlIZHCgYZg6DOsA4YviYfnUrqeWl5RFaVEeAVGACS/lxP3cn8f0se7K7qcwgVwttsv7eVjwTlE8lpENANBPnsTfcz+Juf3cUIYTdQMjEgdIPtVBUxpwOK0X+0v0zOB1SyVf3c1SoXmvM9/iWXUefUstA2FZL+6ZCsM3RYkcixe88V7YtZE3d1gyma/OtvZ6M+MQRe2LRU/Cv6wkuriDcpymQ6ns+UBYRsfvh7Uu/HaQYMS2qYn4FvfLxpUkDQvYhI/SalQ+jJA8RaUxJXkKnG+Oe5ecAHgcifSes6j6POgh6jB1OVtW0rLR+/F81eSlg8j5feeOaoG+pXVXseBnYR3NJo6AhhSBooxftNJs8iF47MPjsvvdiutgiO3gdswEQ/l7SDq5w7PapMqb+PBCyt6D9FTsQe19kgihBdvw7IQJJWICO/YOOxXy9ykF5B2As4WxQhYfgC/JjzPiUXDQTXxK6urUJ5+twgPbugPgthX6/yejz9SDpVDkF2vdKOp/XelaPKfhDBJBl10p3bu36sV+Az8XsJLiyOO+HLcVAo2k2/4SpuZCuFC4+By0W0+/yAQl9qgoTDzBcS652GlalLp8FyJUQGa+63XHugo X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:SY4PR01MB6979.ausprd01.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230028)(4636009)(396003)(346002)(376002)(366004)(136003)(39860400002)(451199021)(31686004)(316002)(786003)(110136005)(66946007)(966005)(66476007)(66556008)(36756003)(186003)(6512007)(53546011)(6506007)(38100700002)(2616005)(83380400001)(66574015)(8676002)(41300700001)(6486002)(5660300002)(8936002)(478600001)(33964004)(166002)(31696002)(30864003)(2906002)(86362001)(45980500001)(43740500002); DIR:OUT; SFP:1101 X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?bk9FRElrSkdaR3hpVkJpNm04c0RxanNSN1FCS0duU0JIRUR6d2EvWUpnZTRP?= =?utf-8?B?cXdMc0NUb1UvYjhVQS9rZ0MxbVI3bUhlK2lHWWNLeE0zNktQaUh4QnBJRTBv?= =?utf-8?B?SCs0b1FmeGtHcGJ1OUJCTGRCZjBnZ1FIU1lJZnJBcmxSQ1BrMXA3V1IycUZE?= =?utf-8?B?OWNvK0MvdUNlaGhUd05NbktqM1RtR2YwRUltOWZ3YjlMQy85S3dQUkJkSUdE?= =?utf-8?B?cjV4N1BBRENmdnNrNVBGRGJSbTNUUHF3VTQ0aFBQbUhLOHh6TEF6SEl5TkRl?= =?utf-8?B?d2tMSkJ3YXhZS3A2WFBsTmdTMExMdWtQRjVYUlZZblhGaGgzd1ZtSyt0M0x1?= =?utf-8?B?azRPR0d3ZlFFYWIrR1dMdlNTNk9NaFM3MjVyVlRyZWJCcGc2amN1UkpGN0lD?= =?utf-8?B?SUIwNWVyMVkxSjVsT3hsSmNyRkZqZEx0SSt1TmIxSk1CVklXejJka2xteFRh?= =?utf-8?B?VlQ1dExEb0lDS2pQQWlpaDBwNEZDeDAxcHpUN1YrWndESmlGdkhicUR4akVi?= =?utf-8?B?YW1Bek9UM2I1MlltVFdLaytlc2hnT2xnd0xpSDBxODNtVHNvWmt6cUc3Z01y?= =?utf-8?B?eUhnWXgzQVZ3MFUxMUpweVFkMXorUHBSeEN5aHF0TFAwZ2t1czgxK3I0dG1i?= =?utf-8?B?ZHVMdTFzTWswVS9xK1NaRHhwYmU4N213TEd4RGhrSlZpcHhueUpzZ0JSbURl?= =?utf-8?B?UXVYa3Y3Qk9PMlRQQWc0elF1cHRsMG5ONEtObkYzY0tYaXk0cXNzbGxjUmt1?= =?utf-8?B?ZmtUSWxrWUZoU0FRcHBDZzlEVThndXZsWWJpaVJnU2s5NUFMMC9uVUZWeWJK?= =?utf-8?B?S2pPVzEwYjZnZDBlY1JMVXV1Q0Nya0Z2SVJEWTYzdnF2VWQ2OHd6Q3JoUWpp?= =?utf-8?B?SFhzNlF5b0JTS01WYnA1MkswWHlFQ0JYeWNTdGhZdWJXM1JsZDNDdVdSZ3ZN?= =?utf-8?B?WGttN3dEcE1UOHl3MDJ6eGFSV2hBZUpoQ1BrU0VqL1JFL3VJOFhwRFBIbk5E?= =?utf-8?B?bWptdmNxdlZSSjZIL0FZbVM1Q2JxMi84VGxySDc1Wk9yRTRGUnFKazMwWStn?= =?utf-8?B?Z29VQXpCY2R6NVF6ZUV5QjRnNXdUSENYckkxdkhQNkdJR0kvNWpuYWNKSkZC?= =?utf-8?B?ekRVSjBKNTVMN0F0OWN6VCtnemxsMFRtUnEzaVdlQkpPK29rL3AzTm1GWndk?= =?utf-8?B?aEZFQkNpR295TEEzanU5T05jRVNTUzBZNSt5V2xNcklpcWM2SHdMOUFkdWFl?= =?utf-8?B?dmJZNFhOVmh4clcxQUpJSlJWcjhkTFRoOWVtMUNqb3VSMjcvZHVyaTR4MVpY?= =?utf-8?B?eTJvM3ZNODdBRzl1UCt0QWFSYXlJTS8yZjNWeXJ4UFdxNVh2VmJXWFRPVW05?= =?utf-8?B?VWxPY2xZaHF6L3M5WHg1SlpadEt4ZXEraUtKVUIxUVRzdkt3aTM3SUN6OUNk?= =?utf-8?B?aGhraDdscGVmY0ZucnRvZzY0R1Flb0VIaStPbVY3OXA2UkQrU2l4M1NXRjll?= =?utf-8?B?ZkVHRUtRT2FjVDdURUJteCs4NkJvYkE2dVBxRTBHY2pRd0xIUWhlOElGYjZk?= =?utf-8?B?MWxCY0toTHJtVVdoWW1QWTFnOGw0YVByb09HTTFmR25lTkZBNlo1ZXNTakpn?= =?utf-8?B?Tm1zTUpJVEFmVW95eThZNkhPU3gwd2JjWFpOSllUWlZkRjdlbXZlWkMzeVJI?= =?utf-8?B?bE5BelZOMUxSTWorZWdUb3ZGakloVTMyU1krcHpOT1RDb2JJbVlsaGt0d0FZ?= =?utf-8?B?QjNkSk9jMWxkWjJhM2dtUDlPWncxQWxaNVF5OHlqa3VDV2hUbFdYVGZNbUpp?= =?utf-8?B?dTV4c0Rqb29nOHRNUEVDTFlNMG9zdU1Md3I5ZytsZ296eEQzVERiTjhFQW1o?= =?utf-8?B?QkErYnIrTGttR2h6VytEZ25zSnpQTmI5dmRXRFF6OXM5Y0dKRHowdlhXQzIx?= =?utf-8?B?ZDF1dzhyUUtFZnFvVFVTWmVuNVJMZmRWb2grTjBsVTlVQlhuRDJ5QnMrWjlK?= =?utf-8?B?UlB1VTBFU1pkQkJPSVZyNnJ6SjN4eThlVnkvaVFlVmlIcGpEM25NVWM1b2dH?= =?utf-8?B?MlkwTmpjbU5tc2pTUG8yOFRGajV3UDhrbkdHaCtadmNPZWR2Q1JQUnhjTi9q?= =?utf-8?B?U0h2WHJXelphSTdqWGpMc3Q2N0VsSGx4eWsrL0tTYUMycTdmTUhnM0tYN3Uv?= =?utf-8?B?cHRZeGFvVC9hL01Pdmx1K29TNXNKZWdpRnNRRlBpb3lSNVFPSEdxWFB0MGFK?= =?utf-8?B?UUxzTTVEVUhRTUFkZ0hVZXlMTm9nPT0=?= X-OriginatorOrg: auckland.ac.nz X-MS-Exchange-CrossTenant-Network-Message-Id: a7683534-dc28-4a5c-1662-08db42bcba90 X-MS-Exchange-CrossTenant-AuthSource: SY4PR01MB6979.ausprd01.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Apr 2023 23:04:15.6280 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: d1b36e95-0d50-42e9-958f-b63fa906beaa X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 5zbJTFQPKAPpsbUnOlSu+aw06+qBcBvtDnbor734O01qA7jTg8MKDjAL/aSdI7IGvo/FVODw+c0g2F7yoEZ1MukpAk23JM8ORa4Uqv32x50= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SY4PR01MB6571 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: auckland.ac.nz Content-Type: multipart/alternative; boundary="------------HKEqu70TacYWAHb0x84rRvLQ" Content-Language: en-US Subject: Re: [Starlink] Starlink in Kiribati X-BeenThere: starlink@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Starlink has bufferbloat. Bad." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Apr 2023 23:04:22 -0000 --------------HKEqu70TacYWAHb0x84rRvLQ Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: quoted-printable On 22/04/2023 4:11 am, Michael Richardson wrote: > > Ulrich Speidel via Starlink wrote: > > All traces (to NZ, Chile, the US, Germany and Japan) exited the SpaceX > > address space in New Zealand to a variety of upstream providers. IP > > addresses on the Starlink side of the trace beyond Dishy are, in=20 > order of > > appearance in the traces: 100.64.0.1=20 > ,=20 > 172.16.249.6, followed by 149.19.109.30=20 > > > if exiting to Hurricane Electric as upstream or 149.19.109.34=20 > =20 > if exiting to > > Kinect. All those addresses have RTT indicating that they are in NZ. > > What are you being NAT44'ed to? Good question - can't answer this right now as Dishy is back in its=20 suitcase in our lab ;-) > Presumably, that IP is in NZ, and you could traceroute/ping back to it=20 > from > elsewhere to be able to subject whatever congestion is beyond the=20 > peering point. Beyond peering point is an interesting thing, too. We spent a day this=20 week on a mountain 5 hours drive south of Auckland running Dishy out of=20 the back of my car, just to find that the one traceroute we took from=20 there shows a routing issue on the Internet side of the POP - HE sent=20 traffic from NZ to Australia, from where it was returned to us, adding=20 over 4000 km of cable to the path each way. Sigh. Ookla tests (to a=20 different machine in a different network in NZ) gave around half the=20 download rates we were observing from the two other locations we visited=20 (and we took 60-180 samples in each location), where that routing issue=20 didn't show up. > > > Lowest RTT seen to the first NZ hop was 65 ms, with values between=20 > 100 and > > 200 ms being the most common. > > > RTTs to the US were mostly in the high 200 ms. > > > 2) The fact that we don't see more than the "usual" number of=20 > Starlink IP > > addresses in the tracreroutes indicates that whatever IP routing may be > > happening on satellites that handle the traffic via ISLs happens at=20 > a tunnel > > layer further down the stack. > > Given Musk's age old tweet that it was "simpler than IPv6", and that=20 > we know > that it's some kind of broadcom SDN chipset, this makes sense. > I wish they had used SR6, and done IPv4 as a service. Indeed, but then who would want to rely on something he said a while ago? > > > 3) The fact that the traffic emerges in New Zealand regardless of globa= l > > destination also indicates that the Starlink network uses a tunnel base= d > > on Dishy location and a nearby gateway but does not attempt to route=20 > to final > > destination at this point in time. > > I'm not surprised about this. > I don't imagine they can world-wide stuff until all the non-ISL birds=20 > have aged out. Well, they do have a complete constellation with lasers, and a polar one=20 that's semi-complete, so in theory they should be able to do this... > > > The 65 ms RTT also tells us a few things. For one, at 4,200 km great=20 > circle > > distance on the ground, the dishy-to-gateway physical path would be=20 > at least > > 5,000 km even if all lined up with a polar orbital plane involved.=20 > That makes > > 10,000 km of RTT path, which translates into about 33 ms of propagation > > RTT. If cross-plane routing were involved here, we'd get a zig-zag=20 > path - so > > roughly 1 1/2 times longer. Makes about 50 ms. In-plane only routing=20 > would > > involve a gateway in Australia (similar length physical path dishy to > > gateway) along with a 2,000 km trans-Tasman cable leg. The 2000 km=20 > cable leg > > would be equivalent to about 20 ms of additional RTT over the 10,000=20 > km space > > RTT path, so that could in principle also work. Quite why everything=20 > would > > emerge in Auckland though in this case would be a mystery to me. > > I think you are saying that the your lowest RTT of 65ms is easily=20 > supported > by physical distances alone? > And that it can't get better. Basically, an observed RTT puts an upper bound on the length of the=20 possible physical path a packet can have taken. On that basis, I'm=20 simply looking at potentially feasible paths here. I'm just observing=20 that if you are landing traffic in Australia where you have multiple=20 gateways and POPs, it makes little sense to then also have a terrestrial=20 tunnel to NZ for the traffic that you are landing and that's destined to=20 other continents. I mean, I'd like Auckland to be the navel of the world, but I somehow=20 doubt that Elon would take the same view ... Then again, at least one of=20 his Silicon Valley mates got NZ citizenship after just two weeks in the=20 country. The guy was going to promote this widely but then somehow that=20 didn't happen and if I correctly remember, it took an Official=20 Information Act request to find out that the guy had become a proud=20 Kiwi. FAIK he hasn't spent much if any time here since becoming a=20 citizen either. So who knows whether Elon might have a secret bolt hole=20 here? Lots of people do... --=20 **************************************************************** Dr. Ulrich Speidel School of Computer Science Room 303S.594 (City Campus) The University of Auckland u.speidel@auckland.ac.nz =20 http://www.cs.auckland.ac.nz/~ulrich/ **************************************************************** --------------HKEqu70TacYWAHb0x84rRvLQ Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
On 22/04/2023 4:11 am, Michael Richardson wrote:
=20
Ulrich Speidel via Starlink <starlink@lists.bufferbloat.n= et> wrote:
> All traces (to NZ, Chile, the US, Germany and Japan) exited the SpaceX
> address space in New Zealand to a variety of upstream providers. IP
> addresses on the Starlink side of the trace beyond Dishy are, in order of
> appearance in the traces: 100.64.0.1, 172.16.249.6, followed by 149.19.109.= 30
> if exiting to Hurricane Electric as upstream or 149.19.109.34 if exiting to > Kinect. All those addresses have RTT indicating that they are in NZ.

What are you being NAT44'ed to?
Good question - can't answer this right now as Dishy is back in its suitcase in our lab ;-)
Presumably, that IP is in NZ, and you could traceroute/ping back to it from
elsewhere to be able to subject whatever congestion is beyond the peering point.
Beyond peering point is an interesting thing, too. We spent a day this week on a mountain 5 hours drive south of Auckland running Dishy out of the back of my car, just to find that the one traceroute we took from there shows a routing issue on the Internet side of the POP - HE sent traffic from NZ to Australia, from where it was returned to us, adding over 4000 km of cable to the path each way. Sigh. Ookla tests (to a different machine in a different network in NZ) gave around half the download rates we were observing from the two other locations we visited (and we took 60-180 samples in each location), where that routing issue didn't show up.

> Lowest RTT seen to the first NZ hop was 65 ms, with values between 100 and
> 200 ms being the most common.

> RTTs to the US were mostly in the high 200 ms.

> 2) The fact that we don't see more than the "usual" nu= mber of Starlink IP
> addresses in the tracreroutes indicates that whatever IP routing may be
> happening on satellites that handle the traffic via ISLs happens at a tunnel
> layer further down the stack.

Given Musk's age old tweet that it was "simpler than IPv6",= and that we know
that it's some kind of broadcom SDN chipset, this makes sense.
I wish they had used SR6, and done IPv4 as a service.
Indeed, but then who would want to rely on something he said a while ago?

> 3) The fact that the traffic emerges in New Zealand regardless of global
> destination also indicates that the Starlink network uses a tunnel based
> on Dishy location and a nearby gateway but does not attempt to route to final
> destination at this point in time.

I'm not surprised about this.
I don't imagine they can world-wide stuff until all the non-ISL birds have aged out.
Well, they do have a complete constellation with lasers, and a polar one that's semi-complete, so in theory they should be able to do this...

> The 65 ms RTT also tells us a few things. For one, at 4,200 km great circle
> distance on the ground, the dishy-to-gateway physical path would be at least
> 5,000 km even if all lined up with a polar orbital plane involved. That makes
> 10,000 km of RTT path, which translates into about 33 ms of propagation
> RTT. If cross-plane routing were involved here, we'd get a zig-zag path - so
> roughly 1 1/2 times longer. Makes about 50 ms. In-plane only routing would
> involve a gateway in Australia (similar length physical path dishy to
> gateway) along with a 2,000 km trans-Tasman cable leg. The 2000 km cable leg
> would be equivalent to about 20 ms of additional RTT over the 10,000 km space
> RTT path, so that could in principle also work. Quite why everything would
> emerge in Auckland though in this case would be a mystery to me.

I think you are saying that the your lowest RTT of 65ms is easily supported
by physical distances alone?
And that it can't get better.

Basically, an observed RTT puts an upper bound on the length of the possible physical path a packet can have taken. On that basis, I'm simply looking at potentially feasible paths here. I'm just observing that if you are landing traffic in Australia where you have multiple gateways and POPs, it makes little sense to then also have a terrestrial tunnel to NZ for the traffic that you are landing and that's destined to other continents.

I mean, I'd like Auckland to be the navel of the world, but I somehow doubt that Elon would take the same view ... Then again, at least one of his Silicon Valley mates got NZ citizenship after just two weeks in the country. The guy was going to promote this widely but then somehow that didn't happen and if I correctly remember, it took an Official Information Act request to find out that the guy had become a proud Kiwi. FAIK he hasn't spent much if any time here since becoming a citizen either. So who knows whether Elon might have a secret bolt hole here? Lots of people do...  


--=20
****************************************************************
Dr. Ulrich Speidel

School of Computer Science

Room 303S.594 (City Campus)

The University of Auckland
u.speidel@auckland.ac.nz=20
http://www.cs.auckland.ac.nz/~ulrich/
****************************************************************
--------------HKEqu70TacYWAHb0x84rRvLQ--