From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from out1-smtp.messagingengine.com (out1-smtp.messagingengine.com [66.111.4.25]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id AA6EB3B29E for ; Wed, 24 May 2023 11:19:18 -0400 (EDT) Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.nyi.internal (Postfix) with ESMTP id 45FC25C00D9; Wed, 24 May 2023 11:19:16 -0400 (EDT) Received: from imap44 ([10.202.2.94]) by compute3.internal (MEProxy); Wed, 24 May 2023 11:19:16 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=handley.org.uk; h=cc:cc:content-type:content-type:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to; s=fm3; t=1684941556; x= 1685027956; bh=zwxMaC7PHGw2GlXgEbl4kNS6rdcSeDUnlKKm+sRckXE=; b=z Nc1b31Oj9rzAixzBzWRpr+Ls5xc1UT55/EvcpNzEULlXjoqEJs97xe0h7cRHPDrT S2jvAZ24Yrul8JfR2d/a7jmNINn2rJbvwcME2rH4+xJgX40WDjxa5VcihCOJF3Xb O+oRs9YIiOdhVYp65SBbepC8ppD44zFY9ZwuiPoN95+vmvQcWT1N5+m5lCaYuU4U cTCNgvDP7gOje4m1hrXlRvh1qUpEhnA+PplbrN1vRcqkxJuu0xB38I9H1ZAmVBEV 3VLraQU9D7x4vfXDEEt6KnKDasWOQXHW+PfeutH3zuq6ixl9XcfOViC4H14/aJ56 VBp7n7GgDuq6OV5oqvNYg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm1; t=1684941556; x=1685027956; bh=zwxMaC7PHGw2G lXgEbl4kNS6rdcSeDUnlKKm+sRckXE=; b=Oi0/R/clTOW/kiYZcvJyvsPh3jNtN LMtIncpP8ddXwBcgev8D3nlCiCi24DsPXdSBf2+iugo5cVcIqF3Yp01FDwz0poJk kzs4TzBcIbpgodvDRYmAwP0dPbgzSN+TGNlqp440L37mewLDfUMGtcQ/OX5Lwmrr VRRnBzD8n63NTVKQMgVVIQUbfDLzNOj60cihslt6n6zfbo/yj7kasiF+hs32Ck2d nSMpNfZ7xqEaItlVXlMVU7TgGOxnFuqtorlqG7YumZCS+t1RozplnPVsL4f6PKdw zlN5rtrDRzltVO1H1r+5tSTNZ1VsmtZwrFURQGDg8ft7DyJ7FYxj5y5nQ== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrfeejhedgkeehucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepofgfggfkjghffffhvfevufgtsegrtderreerredtnecuhfhrohhmpedfofgr rhhkucfjrghnughlvgihfdcuoehmrghrkheshhgrnhgulhgvhidrohhrghdruhhkqeenuc ggtffrrghtthgvrhhnpeeljeevlefgfeevgffgvefgudekheelveehteelgfejudeiueek ueeifeeitdffheenucffohhmrghinhepghhithhhuhgsrdgtohhmnecuvehluhhsthgvrh fuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepmhgrrhhksehhrghnughlvgih rdhorhhgrdhukh X-ME-Proxy: Feedback-ID: i4d08408c:Fastmail Received: by mailuser.nyi.internal (Postfix, from userid 501) id A400E36A0073; Wed, 24 May 2023 11:19:15 -0400 (EDT) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.9.0-alpha0-441-ga3ab13cd6d-fm-20230517.001-ga3ab13cd Mime-Version: 1.0 Message-Id: <624969fc-29ee-4fcf-963a-34afa95b6bc2@app.fastmail.com> In-Reply-To: <1c9df33b-e964-f531-7326-1a11b159e6a7@auckland.ac.nz> References: <0no84q43-s4n6-45n8-50or-12o3rq104n99@ynat.uz> <48b00469-0dbb-54c4-bedb-3aecbf714a1a@auckland.ac.nz> <728orr66-1432-751p-263q-sqopr12s20sq@ynat.uz> <077e6ad1-d7cc-2d57-39f8-e9646bea32a5@auckland.ac.nz> <09552rq0-0n24-0pqo-4085-n918r0n71138@ynat.uz> <9q29o7n2-69rs-3os1-s93q-0795262qs3o1@ynat.uz> <1c9df33b-e964-f531-7326-1a11b159e6a7@auckland.ac.nz> Date: Wed, 24 May 2023 16:18:50 +0100 From: "Mark Handley" To: "Ulrich Speidel" Cc: "starlink@lists.bufferbloat.net" Content-Type: multipart/alternative; boundary=b8e2989b2f224934aa1b3f922d7cdb8b Subject: Re: [Starlink] Starlink hidden buffers X-BeenThere: starlink@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Starlink has bufferbloat. Bad." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 May 2023 15:19:18 -0000 --b8e2989b2f224934aa1b3f922d7cdb8b Content-Type: text/plain On Wed, 24 May 2023, at 1:55 PM, Ulrich Speidel via Starlink wrote: > > Dishy tracks most satellites for significantly less than 15 minutes, and for a relatively small part of their orbit. Let me explain: > > > > > This is an obstruction map obtained with starlink-grpc-tools (https://github.com/sparky8512/starlink-grpc-tools). The way to read this is in polar coordinates: The centre of the image is the dishy boresight (direction of surface normal), distance from the centre is elevation measured as an angle from the surface normal, and direction from the centre is essentially the azimuth - top is north, left is west, bottom is south, and right is east. The white tracks are the satellites dishy uses, and a graph like this gets built up over time, one track at a time. Notice how short the tracks are - they don't follow the satellite for long - typically under a minute. The red bits are satellites getting obscured by the edge of our roof. > > I've also attached a time lapse movie of how one of these graphs builds up - if I correctly remember (the script is on another machine), one frame in the video corresponds to 5 seconds. > > Conclusion: latency change from tracking one satellite is smaller than the latency difference as you jump between satellites. You could be looking at several 100 km of path difference here. In an instant. Even that, at 300,000 km/s of propagation speed, is only in the order of maybe 1 ms or so - peanuts compared to the RTTs in the dozens of ms that we're seeing. But if you get thrown from one queue onto another as you get handed over - what does that do to the remote TCP stack that's serving you? > Interesting video. From eyeballing it, it seems that when it changes satellite, it's most often changing between satellites that are a similar distance from boresight. When it does this, the difference in propogation delay from dishy to satellite will be minimal. It's possible it's even switching when the latency matches - I can't really tell from the video. Of course you can't tell from just one end of the connection whether starlink is switching satellite just when overall ground-to-ground path latency of the current path drops below the path latency of the next path. For that we'd need to see what happened at the groundstation too. But if you were trying to optimize things to minimize reordering, you might try something like this. As you point out, you've still got variable uplink queue sizes to handle as you switch, but there's no fundamental reason why path switches *always* need to result in latency discontinuities. If you did decide to switch when the underlying path latency matches, and thinking more about those uplink queues: when you switch a path from a smaller uplink queue (at a groundstation) to a larger one, there's no reordering, so TCP should be happy(ish). When switching from a larger uplink queue to a smaller one, you can cause reordering, but it's easy enough to hide by adding an earliest release time to any new packets (based on the last time a packet from that flow was (or will be) last sent on the old path), and not release the packets from the new queue to send to the satellite before that time. I've no idea if anyone cares enough to implement such a scheme though. Not saying any of this is what Starlink does - just idle speculation as to how you might minimize reordering if it was enough of a problem. And of course I'm ignoring any queues in satellites... Cheers, Mark --b8e2989b2f224934aa1b3f922d7cdb8b Content-Type: multipart/related; boundary=01a05916f10b4c6d9f26b36a505a8518; type="text/html" --01a05916f10b4c6d9f26b36a505a8518 Content-Type: text/html Content-Transfer-Encoding: quoted-printable

=
On Wed, 24 May 2023, at 1:55 PM, Ulrich Speidel via Starl= ink wrote:

Dishy tracks most satellites f= or significantly less than 15 minutes, and for a relatively small part of their orbit. Let me explain:



This is an obstruction= map obtained with starlink-grpc-tools (https://github.com/sparky8512/starlink-g= rpc-tools). The way to read this is in polar coordinates: The centre of the image is the dishy boresight (direction of surface normal), distance from the centre is elevation measured as an angle from the surface normal, and direction from the centre is essentially the azimuth - top is north, left is west, bottom is south, and right is east. The white tracks are the satellites dishy uses, and a graph like this gets built up over time, one track at a time. Notice how short the tracks are - they don't follow the satellite for long - typically under a minute. The red bits are satellites getting obscured by the edge of our roof.

I've also attached a time= lapse movie of how one of these graphs builds up - if I correctly remember (the script is on another machine), one frame in the video corresponds to 5 seconds.

=

Conclusion: latency change from tracking one satellite is smaller than the latency difference as you jump between satellites. You could be looking at several 100 km of path difference here. In an instant. Even that, at 300,000 km/s of propagation speed, is only in the order of maybe 1 ms or so - peanuts compared to the RTTs in the dozens of ms that we're seeing. But if you get thrown from one queue onto another as you get handed over - what does that do to the remote TCP stack that's serving you? 


Interesting video.  From eyeballing it, it see= ms that when it changes satellite, it's most often changing between sate= llites that are a similar distance from boresight.  When it does th= is, the difference in propogation delay from dishy to satellite will be = minimal.  It's possible it's even switching when the latency matche= s - I can't really tell from the video. 

<= div>Of course you can't tell from just one end of the connection whether= starlink is switching satellite just when overall ground-to-ground path= latency of the current path drops below the path latency of the next pa= th.  For that we'd need to see what happened at the groundstation t= oo.  But if you were trying to optimize things to minimize reorderi= ng, you might try something like this.  As you point out, you've st= ill got variable uplink queue sizes to handle as you switch, but there's= no fundamental reason why path switches *always* need to result in late= ncy discontinuities. 

If you did deci= de to switch when the underlying path latency matches, and thinking more= about those uplink queues: when you switch a path from a smaller uplink= queue (at a groundstation) to a larger one, there's no reordering, so T= CP should be happy(ish).  When switching from a larger uplink queue= to a smaller one, you can cause reordering, but it's easy enough to hid= e by adding an earliest release time to any new packets (based on the la= st time a packet from that flow was (or will be) last sent on the old pa= th), and not release the packets from the new queue to send to the satel= lite before that time.  I've no idea if anyone cares enough to impl= ement such a scheme though.

Not saying any = of this is what Starlink does - just idle speculation as to how you migh= t minimize reordering if it was enough of a problem.  And of course= I'm ignoring any queues in satellites...

C= heers,
Mark

--01a05916f10b4c6d9f26b36a505a8518 Content-Id: Content-Disposition: inline; filename="obstructions_pos2_762.png" Content-Type: image/png; name="obstructions_pos2_762.png" Content-Transfer-Encoding: BASE64 iVBORw0KGgoAAAANSUhEUgAAAHsAAAB7CAIAAADbpWgoAAAE0ElEQVR4nO2Z7VXcMBREVQdt UAdtUAdt0EbqSBvU4UAOxxjNvNGzpd2Vd+f+smV9jK6eTQilGGOMMcYYY4wxxhhjjDHGGGOM McYYY4wxxhhjjDHGGGOMMcacluU/t05xTqg7LXR9aukNhNxte1J3+a3e9r/RNitlaF/PVtiB 0W6PhbZctYvPRc8r8hBnQF//7W1UrXlleHjW/VOteACi8DOOolOkBzliT3NT6Vsbty2VryJf i2iJ7S1Kv3/d0Z4zCmh57lqXxqCT3NUxRO81rXoxqmfFAp+pUWtNh6hxpBrVs6J+RF+ye5Ae fT209AvpXjtggOTYE4Bli9KrWyz2XcvR/kvwTcP+Z5VOSxXf3MyG8wpET3G0OORkxkWpRtUX VX1zbOZpFGwdtXe56aA7ocVePe3R2nS0bN65TGmL85gO3FKm3jNlJU4leWBVaWcizW48KvDC vh6FlV5+CdoezbAAGAnnn113af0vEl7j/pPSxer6PGi8ZtqpEQapX7Hnw6vjDOKAaeD88U9B tZNtY4m9Y8+e1WmLlq4jHc5zcZoScZ/DCwqXw3gFTiITfgowjaiR5m4vlwrjaehGpoBmojVC S+lYNXVaSEqnIW/MahDbo2Por6BOBaKWTyAdA1XbwG7UO86pV+wMjJGaZ3B4xfFo6bRPtQc6 Vqx1PGrw6752PbV0fUt1F1nvtNuQtIW9edg4nfRIVrWHbeMkugszHt0ODDAALZ3usDk2WmJU 1NKqZc7fv/0ZxrBAOa+34gwGlk9meKS7SPu/cr6/L29vPTmHQZVVubGx03IzgO6QqQaS/+Vl VOZe6JboI7qZfvtaOs0QRRJlsTw9zfEt/w8WCD6Keg6p96T0SHdh1UCk9wcdCAbV+8T2UQHE 06TuaJ5lKum0OlS9DK3x7ZzJeAeMl6mMlz3So+tRGaJGNBupD+efSnoVN5KOpTTKOMqikSLd dAhZZUjWY+gdavt0+OEYIgxm6zR+YzAZ3UZJHMDhAPSYd+luDp8ImozKLcEbcGBvOOFh3ZgK V5kOvb0iD6AagjNXF3QJ4V13jobQRecCPWb2qSfJ3FJxuwIsrBSilqkR+xTqo/boKU5LG4Vu YfxMRPvU+y9Qy9HMooWeh9AtwswOcbFpPwadH59Wj0q8bvUIk59S+hJ8IveKFoq3k1erl+BT o3uS85jnTxBNqj1E28t4p64zq+M1jbcExfF18edPl4Vrwktmj2g6Kr80ZsiEJI23/YU+CWra 3pDGQDq9zSyd6Sxi/DQ+Py9PT2OkXI6qaqptYHukuJqnGpjMgP1x8mja5fV1or+x5aEGt48K 1GOm/7aRLpe5jUZ9tby9fRo/uukbgQWL9qMLMaT8FkRn0/PQqNuL5ePjTD8zV5rGqWIxfNte Ytd66WTgSwi5EhnpBZRVw0uirpsLCZt34nolaSHac8b1trNYsbCj/dXj83vy/n5ZHddB+K1E L/BxqNqbh4QT4inSYN8tLy+f/ygcsuvbQ/1S79WQaKpqkupRNOdjgRaS0pvimrX/oERmm7J2 eRdDHu4waDnnvRyQFXnfn/3MVNJ14Q8p0ocrbcqXgI+Pn+ug8P1xuCBU+uo96n/ViHdJVM72 e1UW4NaJHgwbN8YYY4wxxhhjjDHGGGOMifkHPxsBnUEAZ/EAAAAASUVORK5CYII= --01a05916f10b4c6d9f26b36a505a8518-- --b8e2989b2f224934aa1b3f922d7cdb8b--