From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from au-smtp-delivery-117.mimecast.com (au-smtp-delivery-117.mimecast.com [103.96.21.117]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id C735B3B29E for ; Wed, 27 Oct 2021 05:34:39 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=auckland.ac.nz; s=mimecast20200506; t=1635327277; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=OaAqAvEmYGPRwja/Nk8uvTaecM0ZZH0bVZzrU3zVgvQ=; b=Vc+gKnkRDbTMCo5y1JL5QI+IoDIJx2tWxOIDtSszeT3+hAUFIV2MBjE3A1l7X0Si2/2KkF ry3Gmn/aF6O1pAqmswhtsQ0FiCeuv3hd4QBGB6PLF2gtQRwrvfmPhkKr+aB4OrdlEhwfYq X2TMZ9I2q7HIfzxh/0w9OAWUVNggOf8= Received: from AUS01-ME3-obe.outbound.protection.outlook.com (mail-me3aus01lp2235.outbound.protection.outlook.com [104.47.71.235]) (Using TLS) by relay.mimecast.com with ESMTP id au-mta-13-obrU1J0iMHeJbVts8u1R9w-1; Wed, 27 Oct 2021 20:34:36 +1100 X-MC-Unique: obrU1J0iMHeJbVts8u1R9w-1 Received: from SY4PR01MB6979.ausprd01.prod.outlook.com (2603:10c6:10:142::13) by SYAPR01MB2574.ausprd01.prod.outlook.com (2603:10c6:1:a::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4649.14; Wed, 27 Oct 2021 09:34:29 +0000 Received: from SY4PR01MB6979.ausprd01.prod.outlook.com ([fe80::f04c:4ca7:9275:e0fa]) by SY4PR01MB6979.ausprd01.prod.outlook.com ([fe80::f04c:4ca7:9275:e0fa%3]) with mapi id 15.20.4649.014; Wed, 27 Oct 2021 09:34:28 +0000 Message-ID: <49b8e33a-2408-fb9b-0701-51302e1ead47@cs.auckland.ac.nz> Date: Wed, 27 Oct 2021 22:34:26 +1300 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Thunderbird/91.2.0 To: David Lang Cc: starlink@lists.bufferbloat.net References: <28034.1635270711@localhost> <33r3s281-7681-o576-9o66-ss827n15r327@ynat.uz> From: Ulrich Speidel In-Reply-To: <33r3s281-7681-o576-9o66-ss827n15r327@ynat.uz> X-ClientProxiedBy: SYBPR01CA0040.ausprd01.prod.outlook.com (2603:10c6:10:4::28) To SY4PR01MB6979.ausprd01.prod.outlook.com (2603:10c6:10:142::13) MIME-Version: 1.0 Received: from [IPV6:2407:7000:a14b:9100:485a:535e:1ac1:965e] (2407:7000:a14b:9100:485a:535e:1ac1:965e) by SYBPR01CA0040.ausprd01.prod.outlook.com (2603:10c6:10:4::28) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4649.14 via Frontend Transport; Wed, 27 Oct 2021 09:34:28 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: e3808486-f7d1-4988-5f8a-08d9992cf8d9 X-MS-TrafficTypeDiagnostic: SYAPR01MB2574: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:580 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0 X-Microsoft-Antispam-Message-Info: r7i9hGbhhgA8ZrpmcfuB7w0+J0g9nUhkuqFpJC5OGWgXXwQq0N8NYF6myFocSF0wr6eFQ1IrGCS2bTFArPtjoHi34CSYi46zJcZSPt7W1v2JCwS9ovp0VGb9TvfY18/kGrt3g0L887Yn0dnATCChorB39lQT8oFiTodnPkgA3DLjvnXrOhX+Kx4WvchNiBfqyqC8F4W32IF6kE0m9OSWZ2mRE3VwBl+IHRVCe1Npv7pYUNa+aIbz2/rcuK9LHdjWCR29ie75QBoM3RufepxDrkEa4MSyBkeiUTZz/100C9sETvH0AfGn2NCVBwWEjsIXss+w3snUziPykXSuiVs9Lcf6YFX9K/Z9QAmLG6vkx/upbCp1I3PMYAIsgmi0eMTa+OK+OaF49zUzOgk8l8iht+skfH4ptPya1Sbdw0t/cld/Qa6e+Ipj0PFbt7+diUtj2kQZMs6mrB5nJALFprJMrPMYv47vmfJhPeOesvEwSUxp3PoFo/QFQnivMmr5nV83ICfjEzTAv4QD5eA7aVRpB3qLFcgjSIEHjcw1VEveUe5EglJulYHfsIW7VvmYq/je//2XqN9iUIS/JJsRUr4HB9jSkDFHF0wZLpTQdGChe9epfmGzyNZUrC0s4QFgL6axNB7QWuZfh4J2Hda/LMe7tK/mBlvhpj2P3lnAaouhw7EtY8ZhHG3AAURVZc1q4Uslli0+K69gZxkvdmkYobzgJJFG8oBfX7HulaPcrl7MR6YKg2wY2HkswLtguNtiKlOMx0ctiXcb9U+RMV0Qk4/PRTeEn/mvK709HaBU3wFrbnbBPoP4wIU+yF17L/az+1dF X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:SY4PR01MB6979.ausprd01.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(366004)(186003)(2616005)(4326008)(5660300002)(2906002)(508600001)(38100700002)(30864003)(66574015)(6916009)(786003)(316002)(53546011)(83170400001)(66946007)(66556008)(6486002)(66476007)(8676002)(31686004)(8936002)(31696002)(83380400001)(966005)(45980500001)(43740500002); DIR:OUT; SFP:1101 X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?QXU5UHp1MGN1dUVCNnZ6dDhNbkptbzJFSFpyRXBRNUozRGhFSGFDYmozeHVz?= =?utf-8?B?a0k1UklaaWhueENSbVZubmtVSHozdDdsa1o0NWZTdjF5ZHpLbEJrOE9LRjdp?= =?utf-8?B?WGRZSi9nWFg5d010VmphODRkbDR5dC9UZFlvUFZIV09SWE5QbWpIRzNHU0Vh?= =?utf-8?B?S1FwbEc2VzdXTjlyN0RORUJSVkpLbnZWays5WHRNT2p6eFNjZlJQMmN2WGZM?= =?utf-8?B?dmZPYjY3bXN0cUhmemZwVVo5cDJzV3lHOXo2Z0RjcFRRTkpMZTl0czREeVFM?= =?utf-8?B?eml4VzFJNmt6N1BSSFZOaEdlbEpGNWR4eXJHYzZpb0N1NDFTSkFSVmJ1UnVN?= =?utf-8?B?TCs2YndkaExlMUhJaWNQcVppN2VpOHl1dnJERjY5QXIvdmxZd2RtM1lWZ0Ni?= =?utf-8?B?WGMySkhwclhzcXR0TTJwdlE3R1A0NTlLREU1VVJrcjJqS0E5MzB1VXpSaWNP?= =?utf-8?B?Rno0RnE4RldDTU9wQytwQ2p5eTlQNFVYNTlnNjdNYXRoV1pNV3YwUUdpRGtv?= =?utf-8?B?ay9hQ3BRa0Y5d09FZlVxQUdqQlJvc0QrWVBkY3V3ZDE0KzlRUWZVLy9aMU5p?= =?utf-8?B?b1JJaDlUQ0paMmNrM1NvOUMyTFRYOFdPcHpSWFJ2OWdyeW02SmpWMUtjbm5O?= =?utf-8?B?MDFSa1pWcHpGL1FOckN6ZGxLWFA5WHR1SDRtekc4S2Y2MDVIYS9JT1J5blZi?= =?utf-8?B?c1ZxbDlsK3NWREw2R3prVFlPczFPN01pbURDQ29lL0owYW5oRWJaaGEzcU0y?= =?utf-8?B?eVR1YjNpYUlmZ0M2aGo4aGZQcXdOeWROTGtheTI0UUs0U1VESEhiQnhJMU9R?= =?utf-8?B?aHYyY0xDRUZCU2dENWdZVU4wdkxrRXlRcVBZcGhtakdYS0tDdzc0VDdMOXpO?= =?utf-8?B?MGU3TVpTNDdrWlBEbDdmRHF4cHU0TnlCMitFU3Y1UC9EZTFpUllXK1QrdlJp?= =?utf-8?B?TGVLK0pveWRZSEJLOXlNdjRWbGRON08xRlNvMFFWZGpDS3JMMzZNenNaOUIz?= =?utf-8?B?UzRWT2RWSUxmYkQ4V2orUTdNcnI4aWh6aUptdUtxVnpha01IK1pKeXFrcWNL?= =?utf-8?B?dWJ3OHJqeStJUTdQSWlTZzk2bStEemdoUFl5OHdpSVJQMDd3VkVTcmZHUlZs?= =?utf-8?B?NXhUMmtTUzd3RFJsbUhiRlllYnFNUlhaeEt2WERwc2dEYUMySXZldkV6YWpT?= =?utf-8?B?Qnk1TWVvQlpIcUU0VnRWaG82OTJqTEFkTW5IUGR2TS9CQmpETlNJNmJpb3dx?= =?utf-8?B?bmRvaHp2MlZ2MzRQOFR2cWE3OEI3aERNNjljMHZQbGdod1ovWXZZdVJFTWoy?= =?utf-8?B?MWtTcjIwMHkrbzhJeCtENDhOVXZXb0kzNVM5OFk3ZTkzeHFUdWZVQjhFU0Rs?= =?utf-8?B?SmVzWEgwcXUwRjlITjdlcmdPVEl3NDMzem9SZEwwaE9FY00zYjFBTk56MC9D?= =?utf-8?B?Y2kwM1JzM0FvbUI1QVQ1a1Evd3ZQQVlzRUlyMVZjNGk2TW1qRXZhd2MxTDlP?= =?utf-8?B?cDdIYk9ZRkZEUytMQnJiSHF3MXZLRlhUTzc2VkxTcCsvalNadTJGSndtQlFP?= =?utf-8?B?cGRGOXBtbUtLeTRVSkdkT0FxYXdJSXNQMFFNbU00N2V2K0QvS3Z4Z0h4Nzlt?= =?utf-8?B?RWFQc1k2TnFVNTlkUlN5NjNzanZUc3RITlJIZ2tQUTF0YlBhckdVYVQrdkVV?= =?utf-8?B?TmJBMDlpVWJqWmF6aVA4MndzQ0xvOXlnVjUvb0lLQmc0dHIxNWRwQUhueFdX?= =?utf-8?B?Y0hibDc4THVWaTFmRGt1c3Q2MitoUmtXUDZielhsTDdLZGlwMUl4d3lqWHNj?= =?utf-8?B?Z21namhDV1k0c0wwVXlCQ1d5YnVsc0VCQWhzSEhUMmN5V1pqZXNlTGk2aytW?= =?utf-8?B?c0Z3ck56WDRKcytXSno1VldHSW1RR0ZMSStkZkJmRllCWDBKNEZZTjBnNGdq?= =?utf-8?B?S25MREhFY1lSTE16YWRyaUpvM2dWb1pYUWZsdVVuY1NBT3dhSmxMa1RxS0Qx?= =?utf-8?B?b3RVOU9hU1p4eUdFQVI1aTB4L1F2alJEODRJQjA1N1A0c1ZNNkcydzRHSUxv?= =?utf-8?B?bjA3NHhMcnNlcGZKRTdGMFJXRzFqaG9MelNlOG01dnVXMlZobWxHY2YwUjVh?= =?utf-8?B?bEd1N0h6c1c1WVR4Z0FGY0VRUXkzUS9LNnJ4a29ZR3RaR1F6aGdYdnBTREt2?= =?utf-8?B?c0IySXZETU4ydExPa3pRbWp0aHJzaVBPTW5UT0xjdWJpeFVwcXVVbm9nS2F0?= =?utf-8?B?d2tIZEtpWndvWDFVZ0dQQjFTS3h0YkJ5dGowTFcyWEwwNjVJUVIxRW9zUTJa?= =?utf-8?B?WEQ2VktpTDducnAyRFRBd1VER2FJSHJZemVHN0FZd09TVnVkY2RzMzEzM2V4?= =?utf-8?Q?rAs3pYqVBZKpQjWM=3D?= X-OriginatorOrg: cs.auckland.ac.nz X-MS-Exchange-CrossTenant-Network-Message-Id: e3808486-f7d1-4988-5f8a-08d9992cf8d9 X-MS-Exchange-CrossTenant-AuthSource: SY4PR01MB6979.ausprd01.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Oct 2021 09:34:28.5049 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: d1b36e95-0d50-42e9-958f-b63fa906beaa X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: RA1V48P3RvPB62UiK41rpyBVX7OEmlNZ8hJsfZNV8B/1EERI/nnYeZLmtqFr1oSQh8nWMAlxEpwwqwAXxQ3dWCpWVZt5NUGK44DmIPhV14Q= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SYAPR01MB2574 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: auckland.ac.nz Content-Language: en-US Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: quoted-printable Subject: Re: [Starlink] thinking about the laser links again X-BeenThere: starlink@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Starlink has bufferbloat. Bad." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Oct 2021 09:34:40 -0000 On 27/10/2021 8:03 pm, David Lang wrote: > On Wed, 27 Oct 2021, Ulrich Speidel wrote: > >> - Get the satellites to work out where stuff needs to be sent. If=20 >> they were to use something like Bellman-Ford here, that would require=20 >> an enormous amount of update traffic. Dijkstra would require complete=20 >> topology information, which should in principle be computable from an=20 >> almanach on the satellites. But it's still a lot of number crunching=20 >> on a bird. Another option would be for each satellite to compute a=20 >> great circle direction to the target cell and then forward to=20 >> whichever neighbour satellite comes closest to being along that route. > > Remember, IP routing doesn't require anybody have a global view of the=20 > network, just knowlege of nearby nodes. Reality check. IP forwarding (as opposed to routing) checks the destination IP address=20 against a set of routes in your routing table, which are essentially=20 destination networks identified by network IP address and netmask. That=20 decides which interface the routers forwards out through (i.e., to which=20 neighbour). At a local level, one can often supply these routes manually=20 as static routes, but at a global level, it still takes the likes of BGP=20 to help with routing, chiefly because the Internet doesn't have a tree=20 structure at the global level. This update with BGP works nicely in a network whose structure never=20 changes, or changes very slowly, so that set of routes can be updated=20 manually or automatically in a cyclic fashion. But in the case of=20 Starlink, much of the network changes continuously. Things that change for you if you are a Starlink satellite: - Cells you can service (and therefore hosts attached to Dishys you can=20 reach). You may be over Brazil now, the Atlantic next, then over the UK,=20 and then over Germany, and so on... So your on-board routing table would=20 need to reflect that. You never spend more than a minutes or so over the=20 same cell. - Ground station gateways you can attach to - similar timing here. - Satellites on neighbouring orbital planes in the same shell that are=20 in the northbound part of their orbit while you're heading south (and=20 vice versa) come within range and disappear again. - Neighbouring satellites in different shells. Things that don't change: - The satellite in front and behind you in the same orbit. - Neighbouring satellites orbiting in the same direction in adjacent=20 orbital planes. These require tracking but are always in range. So, if we're a satellite, part of our routing table changes every few=20 seconds because some node that was a neighbour no longer is, and a node=20 that wasn't now becomes a neighbour. And because nothing in our=20 constellation-spawned network resembles a tree structure, we'd need=20 something like BGP to propagate that routing information through the=20 network. Because we may be the only satellite that can currently serve a=20 particular destination host, that information needs to propagate through=20 the network, so other nodes in the network need to know that they need=20 to send traffic to that destination to us. I say "something like BGP"=20 here as the network is proprietary, so doesn't have to stick to any open=20 standards. Now our neighbours are the first that need to know that we're the new=20 kid on the block to talk directly to Dave's Dishy, and sending them an=20 update every few seconds wouldn't be too onerous. But the problem is=20 that our neighbours, too, are on the move, so we're basically forced to=20 propagate our update (and everyone else's) throughout the entire network=20 of thousands of satellites. Moreover, if we consider the fact that along=20 a forwarding path of a dozen satellites or more between, say Tokyo and=20 New York, all satellites are on the move, then losing only one link on=20 that path can render a route completely infeasible. And while our=20 individual satellite's neighbourhood changes every few seconds, a path=20 like this might change on a sub-second scale. So suppose we ignore anything we know about the constellation geometry=20 and we have N satellites. Say we want to use IP routing still with BGP.=20 Then each satellite needs to be its own Autonomous System. That boils=20 down to telling each satellite in the constellation about the=20 connectivity state of essentially every other satellite (or plane).=20 That's an O(N^2) problem, so not very scaleable, especially if we need=20 to do this every few milliseconds (with the required intervals also=20 decreasing with N). So conventional IP routing doesn't work here, basically= . > > So the satellite needs to know what are good next hops to send traffic=20 > to that's headed in different directions, not global knowledge of the=20 > entire constellation. True, but that requires a knowledge of direction and a (tunnel) comms=20 layer below the Internet-wide IP routing. And the question is how that's=20 best designed. > > This doesn't require that all satellites be able to send in all=20 > directions. with lots of satellites in range, they could cycle between=20 > the first sending north, the second east, the third south, the fourth=20 > west... > Yes. That's what I was hinting at with the "great circle" option. > A key question to me is how many lasers, how fast they can be=20 > redirected to different satellites, and how many different satellites=20 > the detectors can receive from at one time. > > I'd hope that there are at least two lasers, so that they can transmit=20 > to the next hop and ack to the receiver without having to spin a laser=20 > 180. That's an interesting question indeed. What I would add in here is also=20 the possibility of using optical switching, where one laser beam may be=20 able to be switched rapidly between different targets. > > It also doesn't require that the satellite send to the nearst node in=20 > that direction (if longer distance traffic can be detected, you would=20 > want to send to a further away next hop to reduce the number of hops) In principle, yes, which would be good for both latency and satellite=20 power budget, but satellites further away will be harder to target with=20 the same data rate, so there's a trade-off here. > They could even have a satellite varient that has more=20 > lasers/detectors and fewer RF antennas to connect to the ground to be=20 > dedicated to routing (especially as they launch satellites to=20 > different altitudes, nodes at higher altitudes could be optimized for=20 > long-distance hops) I'd consider this not so likely - the system is complex enough as it is,=20 and I doubt they'll want to add that complexity. > > With only 'nearby' node knowledge needed, I could see low-power=20 > omnidirectional RF beacons being used for nodes to advertise their=20 > existance and routing config to the network. I expect that if they=20 > were doing this, that it would show up in their FCC filings and=20 > everyone would be talking about it, but it's a thought for future=20 > versions. I'm speculating here, but from an engineering point of view it makes=20 most sense to go with an almanach approach to constellation=20 configuration, i.e., let each bird know where all other birds are at any=20 one time by providing their orbital parameters to every satellite (slow=20 rate of change and easy O(N) computation on the satellite). Dishys have=20 a built-in GPS receiver, so can basically figure out which cell they're=20 in (once movement between cells is allowed). Currently, I suspect that=20 Starlink run a protocol where each bird knows which cell it's=20 responsible for at any one time, and where the bird beacons to Dishys=20 "Cell X, it's me, bird Y" and where the Dishys then get orbital=20 parameters either from the bird or an almanach to align their phased=20 arrays to the bird. > > All of this requies some way to tell from the packet where on earth=20 > (or in space) the destination is. If you tunnel over IPv6, you can=20 > have enough IPs to allocate them based on lat/long/ground/orbit so=20 > that the routing nodes (which know where they are) can know what=20 > direction to send things.=20 That's indeed an option, although you'd still need a way of reverse=20 mapping: If I have an IPv6 address I know to be in cell X in Ontario,=20 which of my peers has a route to there? > Given the existance of mobile endpoints (vehicles, aircraft,=20 > spacecraft) this starts sounding like the mobile IP problem (how can=20 > you continue a persistant connection when moving from network to=20 > network), but since all the entry/exit points are under their control,=20 > you can tunnel around and have a protocol to reconfigure the tunnels=20 > as endpoints move. Indeed, and that tunnel configuration protocol is where the challenge=20 lies ;-) For mobile networks, this task is a little easier as the network between=20 the entry & exit points is more or less static. Plus, mobile nodes move=20 at most at hundreds of miles per hour, not tens of thousands. BTW: Starlink will probably want to avoid ground relays like the plague.=20 The bulk of their customers won't be stock brokers, and their need will=20 be for bandwidth rather than minimal latency. The bottlenecks here are=20 between ground and birds, and between birds. Basically, the nominal=20 capacity of each Starlink bird is purportedly somewhere around the 20=20 Gb/s mark in the currently deployed generation. Now, as a rule of thumb,=20 the average Internet user in the world last year used a continuous=20 average of 1 Mb/s of data. If we ignore for a moment that connections=20 are bi-directional and that there is such a thing as peak time, then=20 that means that under the current bent pipe model, each Starlink=20 satellite is restricted to at most 20k users. If they want to beat=20 existing providers on the ground that already offer a few Mb/s, they=20 need to slash that by at least a factor of 5, again being really=20 conservative here. Now those ground providers do have one competitive advantage, and that=20 is that while their lines are long and low bandwidth, they're not=20 shared, at least not to the closest CDN server farms, where 70% or so of=20 today's traffic originates. If two Starlink customers want to watch the=20 same cat video on FB, the video has to traverse the satellite medium=20 twice, whereas if two farmers in the Midwest want to do the same on=20 Verizon, they get two copies off their local CDN and you don't get as=20 much as a blip on regional or international fibre cables (which=20 generally run at rates > 20 Gb/s these days). So if Starlink want to keep customers happy with dozens of Mb/s on=20 demand, each bird will at most be able to handle a few hundred active=20 customers at a time. Any use of multiple ground relays along a path=20 would cut that number substantially. --=20 **************************************************************** Dr. Ulrich Speidel School of Computer Science Room 303S.594 (City Campus) Ph: (+64-9)-373-7599 ext. 85282 The University of Auckland ulrich@cs.auckland.ac.nz http://www.cs.auckland.ac.nz/~ulrich/ ****************************************************************