From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mout.gmx.net (mout.gmx.net [212.227.15.15]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id CC3423B2A4 for ; Sat, 13 May 2023 07:20:30 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.de; s=s31663417; t=1683976824; i=moeller0@gmx.de; bh=MqhRciQyOLHhSznySo70U0AHETZnMwIWNj+x4MQJPbI=; h=X-UI-Sender-Class:Date:From:To:Subject:In-Reply-To:References; b=ODT2d9gXh0unXAj5Dg8g58viB84zb31BhQXvUeKkfqrslGdYYfUzxXTMGX1gbzi+Q DbK3DuNqlgLATrAiEUDIquKdvNwQ7gKvtyjfbqUSy2wHawBJXYxkTliK5BAIfEP7AK wMGQ2REOnuF9ekK6J3LtpBfSYxzaqlEr2q92n0CrYskGUbbBzgxneNn2d1m8zBxza3 u/ZSU95jAdoEVQFdimimZBlFyIiUT5ZZAi4j/cZzmORV/0OpBGkgvyslRqwkrUJM4O 4IlXhk0k3Dp2FjuvlGZ/RXJf+yC5FOwpjjSM5yw83NYC7HDPQJEstLJVyc2IByGw+s O8HupLRmsaRxw== X-UI-Sender-Class: 724b4f7f-cbec-4199-ad4e-598c01a50d3a Received: from [127.0.0.1] ([80.187.110.55]) by mail.gmx.net (mrgmx004 [212.227.17.190]) with ESMTPSA (Nemesis) id 1M1HZo-1pv3D33mME-002pZB; Sat, 13 May 2023 13:20:24 +0200 Date: Sat, 13 May 2023 13:20:19 +0200 From: Sebastian Moeller To: Ulrich Speidel , Ulrich Speidel via Starlink , "starlink@lists.bufferbloat.net" User-Agent: K-9 Mail for Android In-Reply-To: References: Message-ID: <13812EAA-BE80-47BA-A28F-637FA01A0ABB@gmx.de> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary=----JYR3ZJAD4NBAZNWERGDWDRJ2EKQ4AC Content-Transfer-Encoding: 7bit X-Provags-ID: V03:K1:nhVutDkbVbc4lGvNw3cSbalY2Yfp/V3Wgr+a2AgDiav6Hp7EvxU 0pZ9xQsLolZIse/W1Am/FhuAJbpVIP5fL9RSi3/XKI2zhiYqpBrfeYpnGGLfnjIdcKUqC0K UCvVUE9rNbDrwxqHLNdk1btd0E8/wgiMzvZJm/dzsyjr3jmQV0fYY91iSzZkUqHTQfjvmIt B0nTfhxcR8WMgDrZM4+aA== X-Spam-Flag: NO UI-OutboundReport: notjunk:1;M01:P0:OQS7UDeXu6I=;jFk69gpmPOMIpxpve2iHEBgUlB7 ofvR+tD9FF55egaTGOxYQCViBmX5LTYAuH2EpiIXDkes86BEdguULOOUtOaYA+XaElWguwVSb 463ni3u7oAns8f2slvW1bLCDIPu6D9Ngbq0HHdXmzxdFcVOPPsE2WmFRww8IGNWIlLlSHdk0I RRkHZI5UP0ZNwWJXAsHimT4Gx7x+Ge2IlWA4ZhSASVDTkETmBLK6xVI0wKeOZnuwj5T65O/jB 17l+IEmNNkZuquvs9bVnTEf9EhBLyzxUqi+TwHbbWgeMi+q8VYDZW/isEXmxf2iDQLvHXAayj Agu3JG5ilTusT/cBr2U5Zs/srQcy0Ji3YJItDLKgkwMYQdBWO/b/ahyK4FcfyXbIw3XFxxAxM ePFhirPJi43p5aa23QSb7FG/bgX+cEYFun2oRCzDiMePnSU/49437N44iqDIQu4xBh5+Q5hwR lMus/GWY/ZKgKRewwQOorTOmTfXO5GlDRCiNMrzwSufMs0+jJGAFSUX1egmNHu+jKkHPrVBhT ND5C3AWVSxR8cWCkPD6qmB+yC55o0MxAX6doEW6/2L0dF2zqhaHCPh+bgRnf+iim6DdQZxzCJ o319wyIGvPn/oSFHOx11+PZKRhaRjdheAPG9VCdU4ZCGE/afAYCQOg18vFy4YfgRSc6W1ekUW dSTd6Ayd1jZo2hBC9wJmSPG+DIvUHfu9sJ/bHPr4Evi+Ot7fWmV/PxwjBZ2lt5uE6QtL6AYU9 CWuy/7y/9+wDw6D3fo/bJbY+HCms+Fjnu+r3JVL9vAHUk01dkfnepWtmnVhdlC6DCW0u1nsp6 B/fKDnk8HjvgVNtOJ5ClaGOlvwXGuMvrIzY5E/B1sr1tf4c8OkN2vYSDCjJ35NiediXLbDQcL UhjOjDnSmYdOVpP4hRHIjtw+ysS11aGDqy5CSW65o0nmUCaeY2j92afbrIHJlHjCgogKYIgEL 47x57lM5GMMR5okwXnO6ZcXTDSM= Subject: Re: [Starlink] Starlink hidden buffers X-BeenThere: starlink@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Starlink has bufferbloat. Bad." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 13 May 2023 11:20:31 -0000 ------JYR3ZJAD4NBAZNWERGDWDRJ2EKQ4AC Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi Ulrich, This situation is not completely different from say a train full of LTE/5G= users moving through a set of cells with already established 'static' user= s, no? On 13 May 2023 12:10:17 CEST, Ulrich Speidel via Starlink wrote: >Here's a bit of a question to you all=2E See what you make of it=2E > >I've been thinking a bit about the latencies we see in the Starlink netwo= rk=2E This is why this list exist (right, Dave?)=2E So what do we know? > >1) We know that RTTs can be in the 100's of ms even in what appear to be = bent-pipe scenarios where the physical one-way path should be well under 30= 00 km, with physical RTT under 20 ms=2E >2) We know from plenty of traceroutes that these RTTs accrue in the Starl= ink network, not between the Starlink handover point (POP) to the Internet= =2E >3) We know that they aren't an artifact of the Starlink WiFi router (our = traceroutes were done through their Ethernet adaptor, which bypasses the ro= uter), so they must be delays on the satellites or the teleports=2E >4) We know that processing delay isn't a huge factor because we also see = RTTs well under 30 ms=2E >5) That leaves queuing delays=2E > >This issue has been known for a while now=2E Starlink have been innovatin= g their heart out around pretty much everything here - and yet, this buffer= bloat issue hasn't changed, despite Dave proposing what appears to be an ea= sy fix compared to a lot of other things they have done=2E So what are we p= ossibly missing here? > >Going back to first principles: The purpose of a buffer on a network devi= ce is to act as a shock absorber against sudden traffic bursts=2E If I want= to size that buffer correctly, I need to know at the very least (paraphras= ing queueing theory here) something about my packet arrival process=2E > >If I look at conventional routers, then that arrival process involves tra= ffic generated by a user population that changes relatively slowly: WiFi us= ers come and go=2E One at a time=2E Computers in a company get turned on an= d off and rebooted, but there are no instantaneous jumps in load - you don'= t suddenly have a hundred users in the middle of watching Netflix turning u= p that weren't there a second ago=2E Most of what we know about Internet tr= affic behaviour is based on this sort of network, and this is what we've de= signed our queuing systems around, right? > >Observation: Starlink potentially breaks that paradigm=2E Why? Imagine a = satellite X handling N users that are located closely together in a fibre-l= ess rural town watching a range of movies=2E Assume that N is relatively la= rge=2E Say these users are currently handled through ground station telepor= t A some distance away to the west (bent pipe with switching or basic routi= ng on the satellite)=2E X is in view of both A and the N users, but with X = being a LEO satellite, that bliss doesn't last=2E Say X is moving to the (s= outh- or north-)east and out of A's range=2E Before connection is lost, the= N users migrate simultaneously to a new satellite Y that has moved into vi= ew of both A and themselves=2E Y is doing so from the west and is also cate= ring to whatever users it can see there, and let's suppose has been using A= for a while already=2E > >The point is that the user load on X and Y from users other than our N fr= iends could be quite different=2E E=2Eg=2E, one of them could be over the o= cean with few users, the other over countryside with a lot of customers=2E = The TCP stacks of our N friends are (hopefully) somewhat adapted to the con= gestion situation on X with their cwnds open to reasonable sizes, but they = are now thrown onto a completely different congestion scenario on Y=2E Simi= larly, say that Y had less than N users before the handover=2E For existing= users on Y, there is now a huge surge of competing traffic that wasn't the= re a second ago - surging far faster than we would expect this to happen in= a conventional network because there is no slow start involved=2E > >This seems to explain the huge jumps you see on Starlink in TCP goodput o= ver time=2E > >But could this be throwing a few spanners into the works in terms of queu= ing? Does it invalidate what we know about queues and queue management? Wou= ld surges like these justify larger buffers? > >--=20 >**************************************************************** >Dr=2E Ulrich Speidel > >School of Computer Science > >Room 303S=2E594 (City Campus) > >The University of Auckland >u=2Espeidel@auckland=2Eac=2Enz >http://www=2Ecs=2Eauckland=2Eac=2Enz/~ulrich/ >**************************************************************** > > > >_______________________________________________ >Starlink mailing list >Starlink@lists=2Ebufferbloat=2Enet >https://lists=2Ebufferbloat=2Enet/listinfo/starlink --=20 Sent from my Android device with K-9 Mail=2E Please excuse my brevity=2E ------JYR3ZJAD4NBAZNWERGDWDRJ2EKQ4AC Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable
Hi Ulrich,

This situation= is not completely different from say a train full of LTE/5G users moving t= hrough a set of cells with already established 'static' users, no?

On 13 May 2023 12:10:17 C= EST, Ulrich Speidel via Starlink <starlink@lists=2Ebufferbloat=2Enet>= wrote:
Here's a bit of a question to you =
all=2E See what you make of it=2E

I've been thinking a bit about the=
 latencies we see in the Starlink network=2E This is why this list exist (r=
ight, Dave?)=2E So what do we know?

1) We know that RTTs can be in t=
he 100's of ms even in what appear to be bent-pipe scenarios where the phys=
ical one-way path should be well under 3000 km, with physical RTT under 20 =
ms=2E
2) We know from plenty of traceroutes that these RTTs accrue in th=
e Starlink network, not between the Starlink handover point (POP) to the In=
ternet=2E
3) We know that they aren't an artifact of the Starlink WiFi r=
outer (our traceroutes were done through their Ethernet adaptor, which bypa=
sses the router), so they must be delays on the satellites or the teleports=
=2E
4) We know that processing delay isn't a huge factor because we also=
 see RTTs well under 30 ms=2E
5) That leaves queuing delays=2E

Th=
is issue has been known for a while now=2E Starlink have been innovating th=
eir heart out around pretty much everything here - and yet, this bufferbloa=
t issue hasn't changed, despite Dave proposing what appears to be an easy f=
ix compared to a lot of other things they have done=2E So what are we possi=
bly missing here?

Going back to first principles: The purpose of a b=
uffer on a network device is to act as a shock absorber against sudden traf=
fic bursts=2E If I want to size that buffer correctly, I need to know at th=
e very least (paraphrasing queueing theory here) something about my packet =
arrival process=2E

If I look at conventional routers, then that arri=
val process involves traffic generated by a user population that changes re=
latively slowly: WiFi users come and go=2E One at a time=2E Computers in a =
company get turned on and off and rebooted, but there are no instantaneous =
jumps in load - you don't suddenly have a hundred users in the middle of wa=
tching Netflix turning up that weren't there a second ago=2E Most of what w=
e know about Internet traffic behaviour is based on this sort of network, a=
nd this is what we've designed our queuing systems around, right?

Ob=
servation: Starlink potentially breaks that paradigm=2E Why? Imagine a sate=
llite X handling N users that are located closely together in a fibre-less =
rural town watching a range of movies=2E Assume that N is relatively large=
=2E Say these users are currently handled through ground station teleport A=
 some distance away to the west (bent pipe with switching or basic routing =
on the satellite)=2E X is in view of both A and the N users, but with X bei=
ng a LEO satellite, that bliss doesn't last=2E Say X is moving to the (sout=
h- or north-)east and out of A's range=2E Before connection is lost, the N =
users migrate simultaneously to a new satellite Y that has moved into view =
of both A and themselves=2E Y is doing so from the west and is also caterin=
g to whatever users it can see there, and let's suppose has been using A fo=
r a while already=2E

The point is that the user load on X and Y from=
 users other than our N friends could be quite different=2E E=2Eg=2E, one o=
f them could be over the ocean with few users, the other over countryside w=
ith a lot of customers=2E The TCP stacks of our N friends are (hopefully) s=
omewhat adapted to the congestion situation on X with their cwnds open to r=
easonable sizes, but they are now thrown onto a completely different conges=
tion scenario on Y=2E Similarly, say that Y had less than N users before th=
e handover=2E For existing users on Y, there is now a huge surge of competi=
ng traffic that wasn't there a second ago - surging far faster than we woul=
d expect this to happen in a conventional network because there is no slow =
start involved=2E

This seems to explain the huge jumps you see on St=
arlink in TCP goodput over time=2E

But could this be throwing a few =
spanners into the works in terms of queuing? Does it invalidate what we kno=
w about queues and queue management? Would surges like these justify larger=
 buffers?

--
Sent from my Android device with K-9 Mail=2E = Please excuse my brevity=2E
------JYR3ZJAD4NBAZNWERGDWDRJ2EKQ4AC--