From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from vsmx002.dclux.xion.oxcs.net (vsmx002.dclux.xion.oxcs.net [185.74.65.108]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 48C753B29E for ; Wed, 9 Jun 2021 08:17:03 -0400 (EDT) Received: from proxy-2.proxy.oxio.ns.xion.oxcs.net (proxy-2.proxy.oxio.ns.xion.oxcs.net [83.61.18.4]) by mx-out.dclux.xion.oxcs.net (Postfix) with SMTP id BB8B08C0B8A; Wed, 9 Jun 2021 12:16:58 +0000 (UTC) Date: Wed, 9 Jun 2021 14:16:50 +0200 From: Mike Puchol To: Nathan Owens , Dave Taht Cc: starlink@lists.bufferbloat.net Message-ID: <901ae36f-c5f1-4b0d-af17-90d29c212f78@Spark> In-Reply-To: References: X-Readdle-Message-ID: 901ae36f-c5f1-4b0d-af17-90d29c212f78@Spark MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="60c0b139_327fac77_312" X-VadeSecure-Status: LEGIT X-VADE-STATUS: LEGIT Subject: Re: [Starlink] dynamically adjusting cake to starlink X-BeenThere: starlink@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Starlink has bufferbloat. Bad." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Jun 2021 12:17:03 -0000 --60c0b139_327fac77_312 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Greetings Dave, I have just been introduced into the world of bufferbloat - something I=E2= =80=99ve had to deal with in my WISP, but never had a proper name for. I=E2= =80=99m an aerospace engineer by origin, so please forgive my quite likel= y blunders in this space=21 As for the topic at hand, I understand the concept - in essence, if we ca= n find what the Dishy - satellite - gateway triumvirate is doing in terms= of capacity and buffering, we can apply that information into our own ro= uter=E2=80=99s buffer/queue strategy. I have taken a quick look at the do= cument, and there is one assumption that is wrong, A=46AIK, which is that= the satellites are =E2=80=9Cbent pipes=E2=80=9D - they actually process = packets, which means they are an active component of buffering and queuin= g in the path (for optical links to work, satellites would necessarily ha= ve to process packets). At any given time, a satellite may have =7E10k cells under its footprint = that it can serve, and GSO exclusions can exclude between 15% at high lat= itudes and 25% near the Equator. It has three arrays for downlinks and on= e array for uplinks, with 16 beams per array. Thus, it can train up to 48= beams towards the ground, and receive from up to 16 locations on the gro= und at any given time. Thus, to cover each cell, there is a physical TDM = element, plus an additional multiplexing element for each terminal in eac= h cell. If we assume a satellite has a capacity of =7E20 Gbps, limited by= the Ka links from satellite to gateway, each spot beam is capable of del= ivering =7E416 Mbps through each spot beam. If we assume symmetrical perf= ormance, the 16 uplink beams from the one panel would =E2=80=9Ctake in=E2= =80=9D =7E416 Mbps per beam. Thus, a satellite is effectively limited to a 75/25 duty cycle between do= wnlink and uplink, as it can paint each cell three times for download, fo= r every time it receives from it. We then get to the killer - how many cells can a satellite effectively se= rve=3F If we take 8000 cells as an average, each downlink spot beam would= need to care for =7E167 cells, and in uplink =7E500 cells. =46ocusing on= uplink, each cell gets =7E2ms of =E2=80=9Cbeam time=E2=80=9D to transmit= , which is ridiculously low. =46rom my tracker simulations, certain areas= are served by around 8 satellites at any given time, so catering for sat= ellite spacing, we could assume 10ms of =E2=80=9Cbeam time=E2=80=9D. Stil= l to low, as this would only allow for =7E170 packets. My current estimate from back of the envelope calculations is that Starli= nk uses cell TDM factors per beam of 1 to 5, depending on served customer= density under the footprint. Thus, worst case, a satellite is giving a c= ell =7E200ms of downlink time and =7E67ms of uplink time. A Dishy termina= l will need to buffer traffic while it gets its turn on a spot beam. Does this make sense, also from observations=3F Best, Mike On Jun 9, 2021, 11:12 +0200, Dave Taht , wrote: > Dear Mike: > > The biggest thing we need is something that can pull the right stats of= f the dishy and dynamically adjust cake (on outbound at least) to have th= e right amount of buffering for the available bandwidth, so as to make fo= r better statistical multiplexing (=46Q) and active queue management (AQM= ) > > It=E2=80=99s pretty simple: in mangled shell script syntax: > > while up, down =3D getstats() > do > tc qdisc change dev eth0 root cake bandwidth =24up > tc qdisc change dev ifb0 root cake bandwidth =24down > done > > Which any router directly in front of the dishy can do (which is what w= e=E2=80=99ve been doing) > > But whatever magic =E2=80=9Cgetstats()=E2=80=9D would need to do is unc= lear from the stats we get out of it, and a better alternative would be f= or the dishy itself and their headends to be doing this with =E2=80=9CBQL= =22 backpressure. > > As for the huge reductions of latency and jitter under working load, an= d a vast improvement in QoE - for what we=E2=80=99ve been able to achieve= thus far, see appendix A here: > > https://docs.google.com/document/d/1rVGC-iNq2NZ0jk4f3IAiVUHz2S9O6P-=463= vVZU2yBYtw/edit=3Fusp=3Dsharing > > We=E2=80=99ve got plenty more data > on uploads and downloads and other forms of traffic (starlink is optimi= zing for ping, only, over ipv6. Sigh)=E2=80=A6 > > =E2=80=A6 and a meeting with some starlink execs at 11AM today. > > I=E2=80=99m pretty sure at this point we will be able to make a massive= improvement in starlink=E2=80=99s network design very quickly, after tha= t meeting. > > > On Jun 8, 2021, at 2:54 PM, Nathan Owens wrote: > > > > I invited Mike, the creator of the site (starlink.sx) to join the lis= t - he=E2=80=99s put a crazy amount of work in to figure out which sats a= re active (with advice from Jonathan McDowell), programming GSO exclusion= bands, etc. His dayjob is in the ISP business. > > > > > On Sat, Jun 5, 2021 at 8:31 PM Darrell Budic wrote: > > > > https://starlink.sx=C2=A0if you have=E2=80=99t seen it yet. You c= an locate yourself, and it will make some educated guesses about which sa= tellite to which ground station you=E2=80=99re using. Interesting to see = the birds change and the links move between ground stations, lots going o= n to make these things work. > > > > =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= > > > > Starlink mailing list > > > > Starlink=40lists.bufferbloat.net > > > > https://lists.bufferbloat.net/listinfo/starlink > > =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F > > Starlink mailing list > > Starlink=40lists.bufferbloat.net > > https://lists.bufferbloat.net/listinfo/starlink > > =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F > Starlink mailing list > Starlink=40lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/starlink --60c0b139_327fac77_312 Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline
Greetings Dave,

I have just been introduced into the world of bufferbloat - something I=E2= =80=99ve had to deal with in my WISP, but never had a proper name for. I=E2= =80=99m an aerospace engineer by origin, so please forgive my quite likel= y blunders in this space=21

As for the topic at hand, I understand the concept - in essence, if we ca= n find what the Dishy - satellite - gateway triumvirate is doing in terms= of capacity and buffering, we can apply that information into our own ro= uter=E2=80=99s buffer/queue strategy. I have taken a quick look at the do= cument, and there is one assumption that is wrong, A=46AIK, which is that= the satellites are =E2=80=9Cbent pipes=E2=80=9D - they actually process = packets, which means they are an active component of buffering and queuin= g in the path (for optical links to work, satellites would necessarily ha= ve to process packets).

At any given time, a satellite may have =7E10k cells under its footprint = that it can serve, and GSO exclusions can exclude between 15% at high lat= itudes and 25% near the Equator. It has three arrays for downlinks and on= e array for uplinks, with 16 beams per array. Thus, it can train up to 48= beams towards the ground, and receive from up to 16 locations on the gro= und at any given time. Thus, to cover each cell, there is a physical TDM = element, plus an additional multiplexing element for each terminal in eac= h cell. If we assume a satellite has a capacity of =7E20 Gbps, limited by= the Ka links from satellite to gateway, each spot beam is capable of del= ivering =7E416 Mbps through each spot beam. If we assume symmetrical perf= ormance, the 16 uplink beams from the one panel would =E2=80=9Ctake in=E2= =80=9D =7E416 Mbps per beam.

Thus, a satellite is effectively limited to a 75/25 duty cycle between do= wnlink and uplink, as it can paint each cell three times for download, fo= r every time it receives from it.

We then get to the killer - how many cells can a satellite effectively se= rve=3F If we take 8000 cells as an average, each downlink spot beam would= need to care for =7E167 cells, and in uplink =7E500 cells. =46ocusing on= uplink, each cell gets =7E2ms of =E2=80=9Cbeam time=E2=80=9D to transmit= , which is ridiculously low. =46rom my tracker simulations, certain areas= are served by around 8 satellites at any given time, so catering for sat= ellite spacing, we could assume 10ms of =E2=80=9Cbeam time=E2=80=9D. Stil= l to low, as this would only allow for =7E170 packets.

My current estimate from back of the envelope calculations is that Starli= nk uses cell TDM factors per beam of 1 to 5, depending on served customer= density under the footprint. Thus, worst case, a satellite is giving a c= ell =7E200ms of downlink time and =7E67ms of uplink time. A Dishy termina= l will need to buffer traffic while it gets its turn on a spot beam.

Does this make sense, also from observations=3F

Best,

Mike
On Jun 9, 2021, 11:12 +0200, Dave T= aht <davet=40teklibre.net>, wrote:
Dear Mike:

The biggest thing we need is something that can pull = the right stats off the dishy and dynamically adjust cake (on outbound at= least) to have the right amount of buffering for the available bandwidth= , so as to make for better statistical multiplexing (=46Q) and active que= ue management (AQM)

It=E2=80=99s pretty simple: in mangled shell script s= yntax:

while up, down =3D getstats()
do
tc qdisc change dev eth0 root cake bandwidth =24up
tc qdisc change dev ifb0 root cake bandwidth =24down<= /div>
done
&=23160;
Which any router directly in front of the dishy can d= o (which is what we=E2=80=99ve been doing)

But whatever magic =E2=80=9Cgetstats()=E2=80=9D would= need to do is unclear from the stats we get out of it, and a better alte= rnative would be for the dishy itself and their headends to be doing this= with =E2=80=9CBQL=22 backpressure.

As for the huge reductions of latency and jitter unde= r working load, and a vast improvement in QoE - for what we=E2=80=99ve be= en able to achieve thus far, see appendix A here:

https://docs.google.com/document/d/1rVGC-iNq2NZ0jk4f3IAiVUHz2S9O6P= -=463vVZU2yBYtw/edit=3Fusp=3Dsharing

We=E2=80=99ve got plenty more data&=23160;
on uploads and downloads and other forms of traffic (= starlink is optimizing for ping, only, over ipv6. Sigh)=E2=80=A6

=E2=80=A6 and a meeting with some starlink execs at 1= 1AM today.&=23160;

I=E2=80=99m pretty sure at this point we will be able= to make a massive improvement in starlink=E2=80=99s network design very = quickly, after that meeting.&=23160;

On Jun 8, 2021, at 2:54 PM, Nathan Owens <nathan=40nathan.io>= wrote:

I invited Mike, the creator of the s= ite (starlink.sx) = to join the list - he=E2=80=99s put a crazy amount of work in to figure o= ut which sats are active (with advice from Jonathan McDowell), programmin= g GSO exclusion bands, etc. His dayjob is in the ISP business.

On Sat, Jun 5, 2021 at 8:= 31 PM Darrell Budic <budic=40onholyground.com> wr= ote:
https://st= arlink.sx&=23160;if you have=E2=80=99t seen it yet. You can locate yo= urself, and it will make some educated guesses about which satellite to w= hich ground station you=E2=80=99re using. Interesting to see the birds ch= ange and the links move between ground stations, lots going on to make th= ese things work.
=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F
Starlink mailing list
Starlink=40lists.bufferbloat.net
https://lists.buffer= bloat.net/listinfo/starlink
=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F
Starlink mailing list
St= arlink=40lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/starlink

=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F
Starlink mailing list
Starlink=40lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/starlink
--60c0b139_327fac77_312--