From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from au-smtp-delivery-117.mimecast.com (au-smtp-delivery-117.mimecast.com [103.96.21.117]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 398E73B2A4 for ; Sat, 13 May 2023 08:16:45 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=auckland.ac.nz; s=mimecast20200506; t=1683980202; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=4AhTcQKg5mOsZyrCmCptICQAAe1huE2eeQe9xSxuqUo=; b=YLvdh5No70Dw+sbbMjzKYwWQWoujYsCL68XiqXKCbJK4TyEgYquODFYUEM+En0IcCYE/E0 wqQISbIfFhfc3LEcwKFOi/0blvi3u4v2yFwTPeBTik321UklOFjBBui++atw8VerhCEfKs kPfGg7Xnp8o2pDmjYBnZC1SCTQOL9/E= Received: from AUS01-SY4-obe.outbound.protection.outlook.com (mail-sy4aus01lp2173.outbound.protection.outlook.com [104.47.71.173]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id au-mta-95-BdL_WL_gP6CkHMVTQwmG1w-1; Sat, 13 May 2023 22:16:39 +1000 X-MC-Unique: BdL_WL_gP6CkHMVTQwmG1w-1 Received: from SY4PR01MB6979.ausprd01.prod.outlook.com (2603:10c6:10:142::13) by MEWPR01MB9101.ausprd01.prod.outlook.com (2603:10c6:220:1f9::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6387.27; Sat, 13 May 2023 12:16:37 +0000 Received: from SY4PR01MB6979.ausprd01.prod.outlook.com ([fe80::68d5:4e6b:745e:197e]) by SY4PR01MB6979.ausprd01.prod.outlook.com ([fe80::68d5:4e6b:745e:197e%7]) with mapi id 15.20.6387.027; Sat, 13 May 2023 12:16:37 +0000 Message-ID: Date: Sun, 14 May 2023 00:16:34 +1200 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.11.0 To: Sebastian Moeller , Ulrich Speidel via Starlink References: <13812EAA-BE80-47BA-A28F-637FA01A0ABB@gmx.de> From: Ulrich Speidel In-Reply-To: <13812EAA-BE80-47BA-A28F-637FA01A0ABB@gmx.de> X-ClientProxiedBy: SY6PR01CA0064.ausprd01.prod.outlook.com (2603:10c6:10:ea::15) To SY4PR01MB6979.ausprd01.prod.outlook.com (2603:10c6:10:142::13) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SY4PR01MB6979:EE_|MEWPR01MB9101:EE_ X-MS-Office365-Filtering-Correlation-Id: acd1bdae-2d2a-485c-4d94-08db53abe64e X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0 X-Microsoft-Antispam-Message-Info: E8cQG2Ak9xAsUE4TtYb56CSLxDJ+EZLGQSesVCQ09zhf7akblLateyOlovgPKA6rKqUXRUAUsJ9dczuNG1loh99kYmcADQWcjls/yORLPT+g2BrB4qTxvbEpwNw8Eoy8nw7M6za3TgIwwjy5ZKXaRB72Dqq1sCDTfAXJQCKEyle7zfuJeYBjv1WNLM0uEinEdTBIbqRUNHOIc651HFBepe0g+PEm6z4FsjhdCy6WqToy7UKBapI2XjeeBOs0QlrZwdZ4L9YIoVtel0eRDVnXRJMFThuk+wZ6WwqhVnMZFbxOL+8gTTIOBR1hTC5J0oQrdD78kLJwY8sLXNEH3eicehHV1e5+teiWsKJalzTCwuVRMxOW7Ex2M685kAToth6CYd9muZ/oDObtTlVAtuE/Y7vSScuHVr5gVuf0zPijddjRI040MEP0LgL6zF5Y+5a1u8QqET7wy6Gm3/WLp9EVstcxdzpimCPs+XNCxLbkOzWTCDN08MhdoKLipOBiKrkrs3K0dq2fwUzdYcV3RybVpshfkonjwtA0aJiIfTW2w6Rh8F8nn3g2JGFx+y60M9v0ahGNiFKqx7V7p+A2NzdlkPM6hQ09njTgmewTjudTMM/C8xL7tcKlTXUWQCbeEb9Peioc+BtmpcvtlS5a8w/Vkg== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:SY4PR01MB6979.ausprd01.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230028)(4636009)(376002)(136003)(346002)(39860400002)(366004)(396003)(451199021)(478600001)(110136005)(66946007)(6666004)(6486002)(966005)(31686004)(38100700002)(166002)(31696002)(86362001)(186003)(6512007)(6506007)(2616005)(66574015)(33964004)(53546011)(36756003)(41300700001)(30864003)(786003)(316002)(5660300002)(8676002)(8936002)(2906002)(66476007)(66556008)(83380400001)(43740500002)(45980500001); DIR:OUT; SFP:1101 X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?aG5jbnMyd2tLdmJMNzJ5RFUrVG11V3hGUllhaEk4M0VDd3VyY2VqUHhsVjBQ?= =?utf-8?B?VWlLTEY4eFdGKzJWeVNCLzljVzMrT0duRkluVXRoL2hUZGhQZmZaV0wrNzNE?= =?utf-8?B?eURZVkYrcU1ERnlWeGRKWU96ZitlZlp1Sy9tM3RZVTVNZ0NnOCt2aHdrYUNt?= =?utf-8?B?WmJnQkZnMXZ1UW1MSGNIY2JFRXpDUER5L3ZGL2w4VEtDeldsQjExRmdoNUw2?= =?utf-8?B?NFJYU1NIT1kycjMra1FNVEVpdGpsVVlhRFNlNFdMNXBiSGVkUlVnbU9vc0Rj?= =?utf-8?B?cEdGU0dOclQzQTBsdXlLOXg3M2Z1RGZBZCtJaFI1Q0h4OGl3ZmhZdmJuWTdm?= =?utf-8?B?MzlHREh0Q2lhSzNUNTZBYmIvSnFRemZ5bjdyUHZ1YXRkNFR4UnU5ZzVLNk1X?= =?utf-8?B?Tmx3Uk1PbFhTVzM4NGZ1OFB4VFRKWG9oK0tEcytCTGx3Q05SMTBRYmpIRkph?= =?utf-8?B?YmE3VE9OVlo4bVFEZXlwV1crVnVWWEhKR0RuWi9qeUkvZjNMbWdsei8rUkRv?= =?utf-8?B?STJ1ckhybjdKYjR4S1AyTERCL1FvWDdRWUx4WGZOOTNzOE95WHhWT0llcys0?= =?utf-8?B?T3ZMTlFBZ0xOTkxYTzBZZE5TdFBsc3FCNUU0UXM4Z1IwOHBJQ1FKMjBkSWZM?= =?utf-8?B?RmMvN3NJaUUvZytMQkw3U0NLZFNrNW9Pekg0eVU4bEtDU2ZmSFp3UjUxVXZL?= =?utf-8?B?cW5RK0w4WDMvODZyRHNrd0tBV1hFWHhiV05lN0JXSmdLSWtnV0FJeERjYTVs?= =?utf-8?B?L0hQdjNIaUNQdmtaWGxNU2tOdWNqWEZKQ1dQaEVoQ0MvTUFlZnJrRHpFSjJ5?= =?utf-8?B?bjhvWlRDWUNid3gxemdnUWVuelVxQU1XNW1Qam4yV2pWQnV4dk1VNzhjdEpy?= =?utf-8?B?aUdxaTVPbU5qa2cwSGpFZGlHUUtFeGdSTHh2WEF3bGxzNFlqUC9kRndYYmVR?= =?utf-8?B?ODd6OWVXdmIyTksveHpmU1hVczBMN1FrOVJwRzV1T0hadVlXNU9qbGJmai9N?= =?utf-8?B?UGYzRVJxOUlHMitPQjhMWUFzd2hnUFc4NmlTaG9YQjNHVzJRZG8zMmxjOHUr?= =?utf-8?B?cGxJRDFqZVhoTDBFZm5QQndiTldlMGNGMXJTOHg3bkxSbVZnWVVFOFViMWEy?= =?utf-8?B?cTZ6RWFlL25Sd2s2elEwQ1pCWEJIYS9NaEdCa3VxaVppNzNHeDMwOXp5N2Zt?= =?utf-8?B?M3ZFYlNZUTNIRWlidXFvNk5HNkIyVGE0Z1hZSWQ1Zmk4Q1RFazlGODI5YjF5?= =?utf-8?B?b1g1ZmhqVTkzaS82aU10RCtEVE9OL3VYWXBEUDdqMVJXWkd4R0k0ZEF1cmNn?= =?utf-8?B?SWUvN0ZYZVZPMWZHVjIrMFVBbGtRTlJnbUFvL0I2VVpiNUdJb1ZwbEttZzR5?= =?utf-8?B?WkFHcm1GWE5rNUQrRzVqeG9oeU5BcWRMZHpsY01Fd2xtZUxsRXRRb1o4eklS?= =?utf-8?B?WHh4d2RLUUhGVnVPbnhWOU5Ca3dHYzhlamVpQkc1SmhtYWdzZ29heHZiRkxS?= =?utf-8?B?Z0l6TlNLR3JLV0EwSWtQeUUzYm9yZVhSU3dBWjZrZkdtT2dzTnpJQU5oYUI5?= =?utf-8?B?ZE9waEg1bkthRWk1K0FVaThzOC9UdjJYTG05a0RYRXBYV3QxVnFwdHhoWW8x?= =?utf-8?B?R0JzUHJEQ3VJVmUyVWhYbndzMGp5ZTAxOTExdG41bUY0cStrN3JoOWxaMk1J?= =?utf-8?B?cGx3RjFGckY0d0hXNHI0WENLRTVMNUg3eEdZaDJESnltMTFUbk9kQ3NVZDFT?= =?utf-8?B?eDhGVmZaQjlYTkgvdzVocUVqMldSWHNFZGN3ZDQrTUhIYWVRejE5U0N1Rkpr?= =?utf-8?B?TUJ2QTZqQ0hXWTBVRjNqWWxmd2RCd0JINkxHbTg3eUMxZHk2TE1jbUFZcHNT?= =?utf-8?B?M2Y3NGdlOGtBeUc4eDF6VkZaWXhKY0ZlZ2xVb3JEQ01mMEVuaEJuUzNvd1pZ?= =?utf-8?B?YVdSVEZERit6YjVXMkkzU3ZrQVNGc2dPb2w1WmxYVFVhcEtXSUFaQklwRzFy?= =?utf-8?B?d2M3RWJlUGZGNThERjk3TnVqY2wwSmtYQUVWeEsrNlJxSktWSDNzcnd2dWoz?= =?utf-8?B?LzZNaFNIR1BaMURRcnhScW56SEIzdHM5dDVYUE9sUno5ZEtqRDloWmlpTm9N?= =?utf-8?B?SiswU3d1TWRBOSs3eG1XWGRZVXNKVk9QMG9rc3czYnhEQXZlbVlET0Q0eVNK?= =?utf-8?B?RlgxSXJmQ1RMN3pnUndFcFBQcHBGMzZtWEZGWS80ck9za2NSK1E1bVFaVFNr?= =?utf-8?B?YnVDT2xsQTJlS2s0cUNobmlncm9nPT0=?= X-OriginatorOrg: auckland.ac.nz X-MS-Exchange-CrossTenant-Network-Message-Id: acd1bdae-2d2a-485c-4d94-08db53abe64e X-MS-Exchange-CrossTenant-AuthSource: SY4PR01MB6979.ausprd01.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 May 2023 12:16:37.2458 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: d1b36e95-0d50-42e9-958f-b63fa906beaa X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: /2GKDf20D4fRei5dsg2OCJEUSeIFWcAu9qmMLOVcQMD1Z4hvlKIzqOBcT1lqp7zO32zIWBwwwAuw9LAvZ+To9AuDMcaCAWSC3wD+gWp2PwU= X-MS-Exchange-Transport-CrossTenantHeadersStamped: MEWPR01MB9101 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: auckland.ac.nz Content-Type: multipart/alternative; boundary="------------Zowrk0SCPzZ0Ntd7PjTMtxQ1" Content-Language: en-US Subject: Re: [Starlink] Starlink hidden buffers X-BeenThere: starlink@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Starlink has bufferbloat. Bad." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 13 May 2023 12:16:45 -0000 --------------Zowrk0SCPzZ0Ntd7PjTMtxQ1 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: quoted-printable Hi Sebastian, Yes and no. But yes, not completely, and I'm not sure whether anyone has=20 ever looked at this, in fact. People who build mobile networks tend to=20 regard their job as complete the moment they can get an IP packet from a=20 mobile device to the Internet and vice versa, and mobile users tend to=20 be a bit more tolerant if things slow down for a moment or three. There are a few differences, though. One is that cells are (or at least=20 can be) fibre connected, and that is something you would do along a=20 high-speed train line. So there is less of a bottleneck than having to=20 use RF for downlinking. I'd also imaging total user numbers to be lower=20 and the bandwidth demand per user to be less (hands up who takes their=20 50" TV onto trains to watch Netflix in HD?). The other is that most=20 places have 3+ networks serving the train line, which brings down user=20 numbers, or you have in-train cells, which communicate with off-train=20 POPs that have no extra users. But yes, good question IMHO! Cheers, Ulrich On 13/05/2023 11:20 pm, Sebastian Moeller wrote: > Hi Ulrich, > > This situation is not completely different from say a train full of=20 > LTE/5G users moving through a set of cells with already established=20 > 'static' users, no? > > > On 13 May 2023 12:10:17 CEST, Ulrich Speidel via Starlink=20 > wrote: > > Here's a bit of a question to you all. See what you make of it. > I've been thinking a bit about the latencies we see in the > Starlink network. This is why this list exist (right, Dave?). So > what do we know? 1) We know that RTTs can be in the 100's of ms > even in what appear to be bent-pipe scenarios where the physical > one-way path should be well under 3000 km, with physical RTT under > 20 ms. 2) We know from plenty of traceroutes that these RTTs > accrue in the Starlink network, not between the Starlink handover > point (POP) to the Internet. 3) We know that they aren't an > artifact of the Starlink WiFi router (our traceroutes were done > through their Ethernet adaptor, which bypasses the router), so > they must be delays on the satellites or the teleports. 4) We know > that processing delay isn't a huge factor because we also see RTTs > well under 30 ms. 5) That leaves queuing delays. This issue has > been known for a while now. Starlink have been innovating their > heart out around pretty much everything here - and yet, this > bufferbloat issue hasn't changed, despite Dave proposing what > appears to be an easy fix compared to a lot of other things they > have done. So what are we possibly missing here? Going back to > first principles: The purpose of a buffer on a network device is > to act as a shock absorber against sudden traffic bursts. If I > want to size that buffer correctly, I need to know at the very > least (paraphrasing queueing theory here) something about my > packet arrival process. If I look at conventional routers, then > that arrival process involves traffic generated by a user > population that changes relatively slowly: WiFi users come and go. > One at a time. Computers in a company get turned on and off and > rebooted, but there are no instantaneous jumps in load - you don't > suddenly have a hundred users in the middle of watching Netflix > turning up that weren't there a second ago. Most of what we know > about Internet traffic behaviour is based on this sort of network, > and this is what we've designed our queuing systems around, right? > Observation: Starlink potentially breaks that paradigm. Why? > Imagine a satellite X handling N users that are located closely > together in a fibre-less rural town watching a range of movies. > Assume that N is relatively large. Say these users are currently > handled through ground station teleport A some distance away to > the west (bent pipe with switching or basic routing on the > satellite). X is in view of both A and the N users, but with X > being a LEO satellite, that bliss doesn't last. Say X is moving to > the (south- or north-)east and out of A's range. Before connection > is lost, the N users migrate simultaneously to a new satellite Y > that has moved into view of both A and themselves. Y is doing so > from the west and is also catering to whatever users it can see > there, and let's suppose has been using A for a while already. The > point is that the user load on X and Y from users other than our N > friends could be quite different. E.g., one of them could be over > the ocean with few users, the other over countryside with a lot of > customers. The TCP stacks of our N friends are (hopefully) > somewhat adapted to the congestion situation on X with their cwnds > open to reasonable sizes, but they are now thrown onto a > completely different congestion scenario on Y. Similarly, say that > Y had less than N users before the handover. For existing users on > Y, there is now a huge surge of competing traffic that wasn't > there a second ago - surging far faster than we would expect this > to happen in a conventional network because there is no slow start > involved. This seems to explain the huge jumps you see on Starlink > in TCP goodput over time. But could this be throwing a few > spanners into the works in terms of queuing? Does it invalidate > what we know about queues and queue management? Would surges like > these justify larger buffers? > > --=20 > Sent from my Android device with K-9 Mail. Please excuse my brevity. --=20 **************************************************************** Dr. Ulrich Speidel School of Computer Science Room 303S.594 (City Campus) The University of Auckland u.speidel@auckland.ac.nz =20 http://www.cs.auckland.ac.nz/~ulrich/ **************************************************************** --------------Zowrk0SCPzZ0Ntd7PjTMtxQ1 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable

Hi Sebastian,

Yes and no. But yes, not completely, and I'm not sure whether anyone has ever looked at this, in fact. People who build mobile networks tend to regard their job as complete the moment they can get an IP packet from a mobile device to the Internet and vice versa, and mobile users tend to be a bit more tolerant if things slow down for a moment or three.

There are a few differences, though. One is that cells are (or at least can be) fibre connected, and that is something you would do along a high-speed train line. So there is less of a bottleneck than having to use RF for downlinking. I'd also imaging total user numbers to be lower and the bandwidth demand per user to be less (hands up who takes their 50" TV onto trains to watch Netflix in HD?). The other is that most places have 3+ networks serving the train line, which brings down user numbers, or you have in-train cells, which communicate with off-train POPs that have no extra users.

But yes, good question IMHO!

Cheers,

Ulrich 

On 13/05/2023 11:20 pm, Sebastian Moeller wrote:
=20
Hi Ulrich,

This situation is not completely different from say a train full of LTE/5G users moving through a set of cells with already established 'static' users, no?


On 13 May 2023 12:10:17 CEST, Ulrich Speidel via Starlink <starlink@lists.bufferbloat.net> wrote:= 
Here's a bit of a questio=
n to you all. See what you make of it.

I've been thinking a bit about the latencies we see in the Starlink network=
. This is why this list exist (right, Dave?). So what do we know?

1) We know that RTTs can be in the 100's of ms even in what appear to be be=
nt-pipe scenarios where the physical one-way path should be well under 3000=
 km, with physical RTT under 20 ms.
2) We know from plenty of traceroutes that these RTTs accrue in the Starlin=
k network, not between the Starlink handover point (POP) to the Internet.
3) We know that they aren't an artifact of the Starlink WiFi router (our tr=
aceroutes were done through their Ethernet adaptor, which bypasses the rout=
er), so they must be delays on the satellites or the teleports.
4) We know that processing delay isn't a huge factor because we also see RT=
Ts well under 30 ms.
5) That leaves queuing delays.

This issue has been known for a while now. Starlink have been innovating th=
eir heart out around pretty much everything here - and yet, this bufferbloa=
t issue hasn't changed, despite Dave proposing what appears to be an easy f=
ix compared to a lot of other things they have done. So what are we possibl=
y missing here?

Going back to first principles: The purpose of a buffer on a network device=
 is to act as a shock absorber against sudden traffic bursts. If I want to =
size that buffer correctly, I need to know at the very least (paraphrasing =
queueing theory here) something about my packet arrival process.

If I look at conventional routers, then that arrival process involves traff=
ic generated by a user population that changes relatively slowly: WiFi user=
s come and go. One at a time. Computers in a company get turned on and off =
and rebooted, but there are no instantaneous jumps in load - you don't sudd=
enly have a hundred users in the middle of watching Netflix turning up that=
 weren't there a second ago. Most of what we know about Internet traffic be=
haviour is based on this sort of network, and this is what we've designed o=
ur queuing systems around, right?

Observation: Starlink potentially breaks that paradigm. Why? Imagine a sate=
llite X handling N users that are located closely together in a fibre-less =
rural town watching a range of movies. Assume that N is relatively large. S=
ay these users are currently handled through ground station teleport A some=
 distance away to the west (bent pipe with switching or basic routing on th=
e satellite). X is in view of both A and the N users, but with X being a LE=
O satellite, that bliss doesn't last. Say X is moving to the (south- or nor=
th-)east and out of A's range. Before connection is lost, the N users migra=
te simultaneously to a new satellite Y that has moved into view of both A a=
nd themselves. Y is doing so from the west and is also catering to whatever=
 users it can see there, and let's suppose has been using A for a while alr=
eady.

The point is that the user load on X and Y from users other than our N frie=
nds could be quite different. E.g., one of them could be over the ocean wit=
h few users, the other over countryside with a lot of customers. The TCP st=
acks of our N friends are (hopefully) somewhat adapted to the congestion si=
tuation on X with their cwnds open to reasonable sizes, but they are now th=
rown onto a completely different congestion scenario on Y. Similarly, say t=
hat Y had less than N users before the handover. For existing users on Y, t=
here is now a huge surge of competing traffic that wasn't there a second ag=
o - surging far faster than we would expect this to happen in a conventiona=
l network because there is no slow start involved.

This seems to explain the huge jumps you see on Starlink in TCP goodput ove=
r time.

But could this be throwing a few spanners into the works in terms of queuin=
g? Does it invalidate what we know about queues and queue management? Would=
 surges like these justify larger buffers?
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
--=20
****************************************************************
Dr. Ulrich Speidel

School of Computer Science

Room 303S.594 (City Campus)

The University of Auckland
u.speidel@auckland.ac.nz=20
http://www.cs.auckland.ac.nz/~ulrich/
****************************************************************
--------------Zowrk0SCPzZ0Ntd7PjTMtxQ1--