From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail.taht.net (mail.taht.net [176.58.107.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 72EDE3B29E for ; Wed, 9 Jun 2021 09:21:55 -0400 (EDT) Received: from smtpclient.apple (unknown [IPv6:2600:380:455c:bb78:e8f6:553:65b3:eaef]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.taht.net (Postfix) with ESMTPSA id E2B05221D8; Wed, 9 Jun 2021 13:21:53 +0000 (UTC) From: Dave Taht Message-Id: Content-Type: multipart/alternative; boundary="Apple-Mail=_55FFB780-21B6-4639-B5E7-6C2864EB71CD" Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.80.0.2.43\)) Date: Wed, 9 Jun 2021 06:21:51 -0700 In-Reply-To: <901ae36f-c5f1-4b0d-af17-90d29c212f78@Spark> Cc: Nathan Owens , starlink@lists.bufferbloat.net To: Mike Puchol References: <901ae36f-c5f1-4b0d-af17-90d29c212f78@Spark> X-Mailer: Apple Mail (2.3654.80.0.2.43) Subject: Re: [Starlink] dynamically adjusting cake to starlink X-BeenThere: starlink@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Starlink has bufferbloat. Bad." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Jun 2021 13:21:56 -0000 --Apple-Mail=_55FFB780-21B6-4639-B5E7-6C2864EB71CD Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 This was a fun talk. When I gave it I thought=E2=80=A6 "If I could only = get *one* MAJOR ISP to update their routers and headends=E2=80=A6" = https://blog.apnic.net/2020/01/22/bufferbloat-may-be-solved-but-its-not-ov= er-yet/ More below: > On Jun 9, 2021, at 5:16 AM, Mike Puchol wrote: >=20 > Greetings Dave, >=20 > I have just been introduced into the world of bufferbloat - something = I=E2=80=99ve had to deal with in my WISP, but never had a proper name = for. I=E2=80=99m an aerospace engineer by origin, so please forgive my = quite likely blunders in this space! >=20 Not just you, by a long shot. > As for the topic at hand, I understand the concept - in essence, if we = can find what the Dishy - satellite - gateway triumvirate is doing in = terms of capacity and buffering, we can apply that information into our = own router=E2=80=99s buffer/queue strategy. I have taken a quick look at = the document, and there is one assumption that is wrong, AFAIK, which is = that the satellites are =E2=80=9Cbent pipes=E2=80=9D - they actually = process packets, which means they are an active component of buffering = and queuing in the path (for optical links to work, satellites would = necessarily have to process packets). Both my other satcom experts were making the assumption it was actually = a bent pipe at this point. I had thought it did packets, simply because = I didn=E2=80=99t grok how they could possibly globally optimize the world solution any = other way.=20 >=20 > At any given time, a satellite may have ~10k cells under its footprint = that it can serve, and GSO exclusions can exclude between 15% at high = latitudes and 25% near the Equator. It has three arrays for downlinks = and one array for uplinks, with 16 beams per array. Thus, it can train = up to 48 beams towards the ground, and receive from up to 16 locations = on the ground at any given time. Thus, to cover each cell, there is a = physical TDM element, plus an additional multiplexing element for each = terminal in each cell. If we assume a satellite has a capacity of ~20 = Gbps, limited by the Ka links from satellite to gateway, each spot beam = is capable of delivering ~416 Mbps through each spot beam. If we assume = symmetrical performance, the 16 uplink beams from the one panel would = =E2=80=9Ctake in=E2=80=9D ~416 Mbps per beam. >=20 > Thus, a satellite is effectively limited to a 75/25 duty cycle between = downlink and uplink, as it can paint each cell three times for download, = for every time it receives from it. >=20 > We then get to the killer - how many cells can a satellite effectively = serve? If we take 8000 cells as an average, each downlink spot beam = would need to care for ~167 cells, and in uplink ~500 cells. Focusing on = uplink, each cell gets ~2ms of =E2=80=9Cbeam time=E2=80=9D to transmit, = which is ridiculously low. =46rom my tracker simulations, certain areas = are served by around 8 satellites at any given time, so catering for = satellite spacing, we could assume 10ms of =E2=80=9Cbeam time=E2=80=9D. = Still to low, as this would only allow for ~170 packets. >=20 > My current estimate from back of the envelope calculations is that = Starlink uses cell TDM factors per beam of 1 to 5, depending on served = customer density under the footprint. Thus, worst case, a satellite is = giving a cell ~200ms of downlink time and ~67ms of uplink time. A Dishy = terminal will need to buffer traffic while it gets its turn on a spot = beam. >=20 > Does this make sense, also from observations? >=20 > Best, >=20 > Mike > On Jun 9, 2021, 11:12 +0200, Dave Taht , wrote: >> Dear Mike: >>=20 >> The biggest thing we need is something that can pull the right stats = off the dishy and dynamically adjust cake (on outbound at least) to have = the right amount of buffering for the available bandwidth, so as to make = for better statistical multiplexing (FQ) and active queue management = (AQM) >>=20 >> It=E2=80=99s pretty simple: in mangled shell script syntax: >>=20 >> while up, down =3D getstats() >> do >> tc qdisc change dev eth0 root cake bandwidth $up >> tc qdisc change dev ifb0 root cake bandwidth $down >> done >> =20 >> Which any router directly in front of the dishy can do (which is what = we=E2=80=99ve been doing) >>=20 >> But whatever magic =E2=80=9Cgetstats()=E2=80=9D would need to do is = unclear from the stats we get out of it, and a better alternative would = be for the dishy itself and their headends to be doing this with =E2=80=9C= BQL" backpressure. >>=20 >> As for the huge reductions of latency and jitter under working load, = and a vast improvement in QoE - for what we=E2=80=99ve been able to = achieve thus far, see appendix A here: >>=20 >> = https://docs.google.com/document/d/1rVGC-iNq2NZ0jk4f3IAiVUHz2S9O6P-F3vVZU2= yBYtw/edit?usp=3Dsharing = >>=20 >> We=E2=80=99ve got plenty more data=20 >> on uploads and downloads and other forms of traffic (starlink is = optimizing for ping, only, over ipv6. Sigh)=E2=80=A6 >>=20 >> =E2=80=A6 and a meeting with some starlink execs at 11AM today.=20 >>=20 >> I=E2=80=99m pretty sure at this point we will be able to make a = massive improvement in starlink=E2=80=99s network design very quickly, = after that meeting.=20 >>=20 >>> On Jun 8, 2021, at 2:54 PM, Nathan Owens > wrote: >>>=20 >>> I invited Mike, the creator of the site (starlink.sx = ) to join the list - he=E2=80=99s put a crazy = amount of work in to figure out which sats are active (with advice from = Jonathan McDowell), programming GSO exclusion bands, etc. His dayjob is = in the ISP business. >>>=20 >>> On Sat, Jun 5, 2021 at 8:31 PM Darrell Budic > wrote: >>> https://starlink.sx if you have=E2=80=99t = seen it yet. You can locate yourself, and it will make some educated = guesses about which satellite to which ground station you=E2=80=99re = using. Interesting to see the birds change and the links move between = ground stations, lots going on to make these things work. >>> _______________________________________________ >>> Starlink mailing list >>> Starlink@lists.bufferbloat.net = >>> https://lists.bufferbloat.net/listinfo/starlink = >>> _______________________________________________ >>> Starlink mailing list >>> Starlink@lists.bufferbloat.net = >>> https://lists.bufferbloat.net/listinfo/starlink >>=20 >> _______________________________________________ >> Starlink mailing list >> Starlink@lists.bufferbloat.net >> https://lists.bufferbloat.net/listinfo/starlink --Apple-Mail=_55FFB780-21B6-4639-B5E7-6C2864EB71CD Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 This = was a fun talk. When I gave it I thought=E2=80=A6 "If I could only =  get *one* MAJOR ISP to update their routers and headends=E2=80=A6"
https://blog.apnic.net/2020/01/22/bufferbloat-may-be-solved-but= -its-not-over-yet/

More below:

On Jun 9, 2021, at 5:16 AM, = Mike Puchol <mike@starlink.sx> wrote:

Greetings Dave,

I have just been introduced into the world of bufferbloat - something = I=E2=80=99ve had to deal with in my WISP, but never had a proper name = for. I=E2=80=99m an aerospace engineer by origin, so please forgive my = quite likely blunders in this space!


Not just you, by a long shot.

As for the topic at hand, I understand the concept - in essence, if we = can find what the Dishy - satellite - gateway triumvirate is doing in = terms of capacity and buffering, we can apply that information into our = own router=E2=80=99s buffer/queue strategy. I have taken a quick look at = the document, and there is one assumption that is wrong, AFAIK, which is = that the satellites are =E2=80=9Cbent pipes=E2=80=9D - they actually = process packets, which means they are an active component of buffering = and queuing in the path (for optical links to work, satellites would = necessarily have to process packets).

Both my other satcom experts were making the assumption = it was actually a bent pipe at this point.  I had thought it did = packets, simply because I didn=E2=80=99t
grok how they could = possibly globally optimize the world solution any other = way. 


At any given time, a satellite may have ~10k cells under its footprint = that it can serve, and GSO exclusions can exclude between 15% at high = latitudes and 25% near the Equator. It has three arrays for downlinks = and one array for uplinks, with 16 beams per array. Thus, it can train = up to 48 beams towards the ground, and receive from up to 16 locations = on the ground at any given time. Thus, to cover each cell, there is a = physical TDM element, plus an additional multiplexing element for each = terminal in each cell. If we assume a satellite has a capacity of ~20 = Gbps, limited by the Ka links from satellite to gateway, each spot beam = is capable of delivering ~416 Mbps through each spot beam. If we assume = symmetrical performance, the 16 uplink beams from the one panel would = =E2=80=9Ctake in=E2=80=9D ~416 Mbps per beam.

Thus, a satellite is effectively limited to a 75/25 duty cycle between = downlink and uplink, as it can paint each cell three times for download, = for every time it receives from it.

We then get to the killer - how many cells can a satellite effectively = serve? If we take 8000 cells as an average, each downlink spot beam = would need to care for ~167 cells, and in uplink ~500 cells. Focusing on = uplink, each cell gets ~2ms of =E2=80=9Cbeam time=E2=80=9D to transmit, = which is ridiculously low. =46rom my tracker simulations, certain areas = are served by around 8 satellites at any given time, so catering for = satellite spacing, we could assume 10ms of =E2=80=9Cbeam time=E2=80=9D. = Still to low, as this would only allow for ~170 packets.

My current estimate from back of the envelope calculations is that = Starlink uses cell TDM factors per beam of 1 to 5, depending on served = customer density under the footprint. Thus, worst case, a satellite is = giving a cell ~200ms of downlink time and ~67ms of uplink time. A Dishy = terminal will need to buffer traffic while it gets its turn on a spot = beam.

Does this make sense, also from observations?

Best,

Mike
On Jun 9, 2021, 11:12 = +0200, Dave Taht <davet@teklibre.net>, wrote:
Dear Mike:

The biggest thing we need is something that can pull the = right stats off the dishy and dynamically adjust cake (on outbound at = least) to have the right amount of buffering for the available = bandwidth, so as to make for better statistical multiplexing (FQ) and = active queue management (AQM)

It=E2=80=99s pretty simple: in mangled shell script = syntax:

while up, down =3D getstats()
do
tc qdisc change dev eth0 root cake bandwidth $up
tc qdisc change dev ifb0 root cake bandwidth $down
done
 
Which any router directly in front of the dishy can do = (which is what we=E2=80=99ve been doing)

But whatever magic =E2=80=9Cgetstats()=E2=80=9D would = need to do is unclear from the stats we get out of it, and a better = alternative would be for the dishy itself and their headends to be doing = this with =E2=80=9CBQL" backpressure.

As for the huge reductions of latency and jitter under = working load, and a vast improvement in QoE - for what we=E2=80=99ve = been able to achieve thus far, see appendix A here:

https://docs.google.com/document/d/1rVGC-iNq2NZ0jk4f3IAiVUHz2S9= O6P-F3vVZU2yBYtw/edit?usp=3Dsharing

We=E2=80=99ve got plenty more data 
on uploads and downloads and other forms of traffic = (starlink is optimizing for ping, only, over ipv6. Sigh)=E2=80=A6

=E2=80=A6 and a meeting with some starlink execs at 11AM = today. 

I=E2=80=99m pretty sure at this point we will be able to = make a massive improvement in starlink=E2=80=99s network design very = quickly, after that meeting. 

On Jun 8, 2021, at 2:54 PM, Nathan Owens <nathan@nathan.io> = wrote:

I invited Mike, the creator of the site (starlink.sx) to join the = list - he=E2=80=99s put a crazy amount of work in to figure out which = sats are active (with advice from Jonathan McDowell), programming GSO = exclusion bands, etc. His dayjob is in the ISP business.

On Sat, Jun 5, 2021 at 8:31 PM = Darrell Budic <budic@onholyground.com> wrote:
https://starlink.sx if you have=E2=80=99t seen it = yet. You can locate yourself, and it will make some educated guesses = about which satellite to which ground station you=E2=80=99re using. = Interesting to see the birds change and the links move between ground = stations, lots going on to make these things work.
_______________________________________________
Starlink mailing list
Starlink@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/starlink
_______________________________________________
Starlink mailing list
Starlink@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/starlink

_______________________________________________
Starlink mailing list
Starlink@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/starlink

= --Apple-Mail=_55FFB780-21B6-4639-B5E7-6C2864EB71CD--