From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from au-smtp-delivery-117.mimecast.com (au-smtp-delivery-117.mimecast.com [103.96.23.117]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 02A613B29E for ; Sat, 13 May 2023 06:10:25 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=auckland.ac.nz; s=mimecast20200506; t=1683972623; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=AVTAioKONXSRpyU+zDoibPYm6CKpK8KLEeVZgP7sItg=; b=UeG0r0dFeHOZFk3/1MzFOJaiC+LS+7sCLvc+WkJWbMTblCK3xEcoBL9lTXiaeNUNNJkiPJ u9WgiSIOiLPutdP0tG8xPCyB8hrYllIsUQ6gwXy/U405moXSVyozj3r2ASwr3ks+WaTKll Ek5uOOGhostJdSVDSu2DBmrKP4xgOp8= Received: from AUS01-SY4-obe.outbound.protection.outlook.com (mail-sy4aus01lp2169.outbound.protection.outlook.com [104.47.71.169]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id au-mta-16-55-NIF9OMde6Z0d18Dewtw-1; Sat, 13 May 2023 20:10:22 +1000 X-MC-Unique: 55-NIF9OMde6Z0d18Dewtw-1 Received: from SY4PR01MB6979.ausprd01.prod.outlook.com (2603:10c6:10:142::13) by SY4PR01MB7237.ausprd01.prod.outlook.com (2603:10c6:10:158::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6387.28; Sat, 13 May 2023 10:10:20 +0000 Received: from SY4PR01MB6979.ausprd01.prod.outlook.com ([fe80::68d5:4e6b:745e:197e]) by SY4PR01MB6979.ausprd01.prod.outlook.com ([fe80::68d5:4e6b:745e:197e%7]) with mapi id 15.20.6387.027; Sat, 13 May 2023 10:10:20 +0000 Message-ID: Date: Sat, 13 May 2023 22:10:17 +1200 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.11.0 To: "starlink@lists.bufferbloat.net" From: Ulrich Speidel X-ClientProxiedBy: SY6PR01CA0090.ausprd01.prod.outlook.com (2603:10c6:10:110::23) To SY4PR01MB6979.ausprd01.prod.outlook.com (2603:10c6:10:142::13) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SY4PR01MB6979:EE_|SY4PR01MB7237:EE_ X-MS-Office365-Filtering-Correlation-Id: 6c79dc0c-014d-4f9d-3003-08db539a41e3 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0 X-Microsoft-Antispam-Message-Info: 8oU75hK7qjS9HPPiD653LOP2UN7PyTZp5AxX9uRD/uP8Bt+xBeMniX7Q8xJEUa9TWNlOvmV2qUoVEO7tfKqezaERhuQzfJ1gGxFiWkZNqN7HQyTwB2/d3yk4+IuGruPe0oZhS65igATWJBkVugdKo3D95KRGpTsO6/k1UziNn0vb8p9Sm21wpjdEq5kNSnMTOZdftKE1Maw6ShYwnJvgXkLeDaQQoWX4CIjdbhe9DJyNruQktPVpk4BLxvzD/f9i9Rb1GjIbquhVijOGs7K2JEL4Ql8jeINXaPrw6++bVqFA4cHNAkkQJ3QJQ4WlDkqQVH3Vo2+zvhrgMlTqnfE+8LYHSGXwIkQl2g+/YtJqvZ2g/1G5ValL73TtWYi434gqEJEM32y9lapySMBI49xygO3MqMX2dv+DJc2p+KP518eJn/02sPF5r5Ih0N5ti0APEMgDZ68r0slakvevHU63tmU0VoIjcid60vrIMpyVm23atu5gDk66GNiehA5U86hFusn91004CzsMhHPaUXrJRQLrpURk5lB7ysKDf9DWW0nYIxmWBFD5rQM+NXsN3KUuJA68pZelUDVkqFeNvIS2JRiIOQTCYEPnetXqEQWiQVKi1GjyPa2H4KttNqJxyYLRDzR7xMB70vBdqVUGeG74Fw== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:SY4PR01MB6979.ausprd01.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230028)(4636009)(366004)(39860400002)(396003)(136003)(376002)(346002)(451199021)(36756003)(86362001)(31696002)(786003)(316002)(66946007)(66556008)(66476007)(6916009)(966005)(478600001)(6486002)(6666004)(8936002)(8676002)(5660300002)(7116003)(41300700001)(2906002)(3480700007)(38100700002)(2616005)(186003)(6512007)(6506007)(83380400001)(66574015)(31686004)(45980500001)(43740500002); DIR:OUT; SFP:1101 X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?Ty9vb1V6VCt3TU9GVGFYOUY3eU9TYmhXMVczRU1Ddm1hQWhrRmZ3bDgxSFJF?= =?utf-8?B?WmJiUytrNTY4bjFzd3RTM0dCWjBzN1pBK1JHTTJjdy9nemdWTjFWUWl3bkNo?= =?utf-8?B?OWJnNitYemNPM2IvVW5sNklxU016R0JReWFrcjFicFl6ckRRN1ZlYkJRN25S?= =?utf-8?B?Wk5mOEdCY1BCWnVZUXJRLzRLSFZuSlNJelczR2xxdEdVRjUvNm1XK00yOE9v?= =?utf-8?B?RHlSODVES0E4cklNaDNmNnNZUEZNSzZTQlU1Uld2Ty8zTXZITDh1RWJIZGJP?= =?utf-8?B?SVNVT3pLUGxzd1hGV0kvazhpMWxmeVJ4OXZXNENrbnpSRm5rU0xhUi82TjNP?= =?utf-8?B?NjhkVjJmVko4eEU0ZTc5UVZvM2pZaFdEbENCQm9QZVM5V0pRNDFpbWszUDQx?= =?utf-8?B?cVlkS2IxRHJUV3pXSmJranlqU3ZsSkNiY1NRajdEdU9wT2tkOVFYM3R2TWFj?= =?utf-8?B?UTNxem53c0JJVEhwbnFoamh3SWttM3hPTytpV0tCVzJ1aEdVSWN3bnFYc3E0?= =?utf-8?B?NnFkdEFkYmRHaVN4TTFJbFlEYzhVbkMxTm83ZlFhb21kK01GQndhL0dybXhL?= =?utf-8?B?bFUyMFN2QVEveUQ2MDhHdWVnaE1YeWo0dGQxMWwyVEFuQm5BS0Rya2IzOHcx?= =?utf-8?B?VUlGR3FBVDEzeWNOM1RVL3BwWkFwTkZITDVTWXpBRG5Yc3RhbDJXY096OHRp?= =?utf-8?B?WUdJUlVvTzdnemp3TTdrTSs2SFZHK1Y0R3dsazRvdFF1VzNWT08wN01LSDZz?= =?utf-8?B?Qk42RnkzN0hQZXBUMmZPeXlCZlF1elhYUUhLTGhDQ1R6QjNKbFJKQjF6YjZE?= =?utf-8?B?RDdzbUsyOXFLUkxURWRHVnlOejZ4cEFseDlVRDcwTGhSUjk4bjJudjVRK1dS?= =?utf-8?B?ZXl6RDZERVFFd3lFTURJSVdUcDArT20yYWpFT0srN3hTMDNtbXYwSHhScXBC?= =?utf-8?B?TnhvOTJHNjZtNlB3VStNZFoxcHFqRnpBY3ZCQU8yK0VIVC9YeE9QMS9Dc0lR?= =?utf-8?B?OEFEeUVnSmpyUVp4WFRGdkJVMk1aMlQ2eVlkNnBwL1VIODU4dTdTQTYydkdz?= =?utf-8?B?VlY3NitTQ1Jwakh0enI1VG1IMENBd2hJbmc3L3FHb2EwdkpkZFYwRForVEls?= =?utf-8?B?cXpuWU1TM0dITjZidWN6Z25jazlob3RjdmRYWXZ0K0NIT1VrVWJOeVVFQnYy?= =?utf-8?B?T1I5MTB3QU1rYVNJeHlzandxekR5ZjY3Y3pBdy9EYlhFdWR6cklOSWpnQjdv?= =?utf-8?B?VjVNY3NmdlgzNVVmM0cyaWtrdjd3cm5yOFR4ZnVvWUt6VmdDcWx4dG5CNk5q?= =?utf-8?B?bGhUQlo4WDFiYVBoMVpOR3R6RitFT0EvejAvZ0lWczh2ZGpScmhNd2llQ1p0?= =?utf-8?B?TW1zOG15Y3NlbG10MHJlR2g2Mk5uSGJGbXMwQzhOR2diVDlUNGFpRHowQllu?= =?utf-8?B?aEV2Sk1JTVRBVExMRnQxVHJBTFFzTlhhZGNPcGpQSy9XWnVwKzJzNUIxZGJj?= =?utf-8?B?S0ptaWcyQ1JhOC9nL2EwdHdyam1qcHZHbmFrZXY3RW1KS1VYRHI1aUphK0lj?= =?utf-8?B?Vkt4cC90VUk1eE9rNkFiN3NjZVduK0NHcXA0KzFHdEt0YUkvcGZEM2RoWHpW?= =?utf-8?B?bXd5Um9sbzFUTUtINHhoUmZNUXNLY21WYUxUSWV4SldmZEN0OHc4MDhWRFB4?= =?utf-8?B?aElwdkRCYTZyQW92bXdkUWVlZUg5TzVpSHdzOEt0ejI4eDBYcytuSmdSTXB4?= =?utf-8?B?azFRVzNmUHRXb3BWUGR2bFdzRldjT05ZL2RMZmFTZ1dhSDlvb1dnNzVLSzFa?= =?utf-8?B?MGVWaHRSa3VjZ2ZCRnN2azltY0lFRExBZmJ0M1dNYVh5YTUyYnZ3SVlrUzNx?= =?utf-8?B?eG9USWxyb1lFcUd0V0FZekJuNWw4RUpIZHF4S0w1SCtUMkxsRDduNjlaVDlS?= =?utf-8?B?ajRtNUlzZDRTTmdSd0ZxaFN4eW1mV3E3aWlnaDhYRUZNQmh5dUEwNGVZWG16?= =?utf-8?B?NmxMQ0lJM21ZcnNuNCtRaXQwL2JDcXZJbVBFaHJvRWpQelhDb2JiWldIUFFo?= =?utf-8?B?Z01KMEp6NE0ycTcwaTRtaWp4aStxMzZZSzJMa0t5NnFsc3JvQWwxZ1pGOUZI?= =?utf-8?B?Vjl0bW5qRWsvSkpOUkRUeE5peDN3MWhxYXdRcEQvcjZCS0ZwRlBoa0FNNUJy?= =?utf-8?B?empKeW1adlk1ZzJWbUI0aHRINTBPNWluMVNIRmFacXRCckpBRmo4ZkRsQ0tr?= =?utf-8?B?K3dreTlvc243eitjalJIcVpDMmRnPT0=?= X-OriginatorOrg: auckland.ac.nz X-MS-Exchange-CrossTenant-Network-Message-Id: 6c79dc0c-014d-4f9d-3003-08db539a41e3 X-MS-Exchange-CrossTenant-AuthSource: SY4PR01MB6979.ausprd01.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 May 2023 10:10:19.9608 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: d1b36e95-0d50-42e9-958f-b63fa906beaa X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 7v/tzw4zZx3lY3hzWtvvpWFp1gQCL867dnB1NUMAeyH5WdXiMDHm8f8MdZoH7AVB2Z719XLSBMsXEbjSMzfBO+J6su0D21WiGXozBPKurmQ= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SY4PR01MB7237 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: auckland.ac.nz Content-Language: en-US Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: quoted-printable Subject: [Starlink] Starlink hidden buffers X-BeenThere: starlink@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Starlink has bufferbloat. Bad." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 13 May 2023 10:10:26 -0000 Here's a bit of a question to you all. See what you make of it. I've been thinking a bit about the latencies we see in the Starlink=20 network. This is why this list exist (right, Dave?). So what do we know? 1) We know that RTTs can be in the 100's of ms even in what appear to be=20 bent-pipe scenarios where the physical one-way path should be well under=20 3000 km, with physical RTT under 20 ms. 2) We know from plenty of traceroutes that these RTTs accrue in the=20 Starlink network, not between the Starlink handover point (POP) to the=20 Internet. 3) We know that they aren't an artifact of the Starlink WiFi router (our=20 traceroutes were done through their Ethernet adaptor, which bypasses the=20 router), so they must be delays on the satellites or the teleports. 4) We know that processing delay isn't a huge factor because we also see=20 RTTs well under 30 ms. 5) That leaves queuing delays. This issue has been known for a while now. Starlink have been innovating=20 their heart out around pretty much everything here - and yet, this=20 bufferbloat issue hasn't changed, despite Dave proposing what appears to=20 be an easy fix compared to a lot of other things they have done. So what=20 are we possibly missing here? Going back to first principles: The purpose of a buffer on a network=20 device is to act as a shock absorber against sudden traffic bursts. If I=20 want to size that buffer correctly, I need to know at the very least=20 (paraphrasing queueing theory here) something about my packet arrival=20 process. If I look at conventional routers, then that arrival process involves=20 traffic generated by a user population that changes relatively slowly:=20 WiFi users come and go. One at a time. Computers in a company get turned=20 on and off and rebooted, but there are no instantaneous jumps in load -=20 you don't suddenly have a hundred users in the middle of watching=20 Netflix turning up that weren't there a second ago. Most of what we know=20 about Internet traffic behaviour is based on this sort of network, and=20 this is what we've designed our queuing systems around, right? Observation: Starlink potentially breaks that paradigm. Why? Imagine a=20 satellite X handling N users that are located closely together in a=20 fibre-less rural town watching a range of movies. Assume that N is=20 relatively large. Say these users are currently handled through ground=20 station teleport A some distance away to the west (bent pipe with=20 switching or basic routing on the satellite). X is in view of both A and=20 the N users, but with X being a LEO satellite, that bliss doesn't last.=20 Say X is moving to the (south- or north-)east and out of A's range.=20 Before connection is lost, the N users migrate simultaneously to a new=20 satellite Y that has moved into view of both A and themselves. Y is=20 doing so from the west and is also catering to whatever users it can see=20 there, and let's suppose has been using A for a while already. The point is that the user load on X and Y from users other than our N=20 friends could be quite different. E.g., one of them could be over the=20 ocean with few users, the other over countryside with a lot of=20 customers. The TCP stacks of our N friends are (hopefully) somewhat=20 adapted to the congestion situation on X with their cwnds open to=20 reasonable sizes, but they are now thrown onto a completely different=20 congestion scenario on Y. Similarly, say that Y had less than N users=20 before the handover. For existing users on Y, there is now a huge surge=20 of competing traffic that wasn't there a second ago - surging far faster=20 than we would expect this to happen in a conventional network because=20 there is no slow start involved. This seems to explain the huge jumps you see on Starlink in TCP goodput=20 over time. But could this be throwing a few spanners into the works in terms of=20 queuing? Does it invalidate what we know about queues and queue=20 management? Would surges like these justify larger buffers? --=20 **************************************************************** Dr. Ulrich Speidel School of Computer Science Room 303S.594 (City Campus) The University of Auckland u.speidel@auckland.ac.nz http://www.cs.auckland.ac.nz/~ulrich/ ****************************************************************