From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-il1-x135.google.com (mail-il1-x135.google.com [IPv6:2607:f8b0:4864:20::135]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id D66203B29E for ; Wed, 9 Jun 2021 11:32:58 -0400 (EDT) Received: by mail-il1-x135.google.com with SMTP id j26so16794992ila.4 for ; Wed, 09 Jun 2021 08:32:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nathan.io; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=TlcSHO0PYUHPy0BBchFGR15kwnEcxtL9LEFFMBual28=; b=IijBJdfyzNP63Ag3mrj/QT9w8INoFzal/NXFWLcOaMLHhZ+6Oo+8dGEyMat3qFYIce HD45eb2G/soALbobpC47iauBxDJt/A8MeyE2IVKqBSUO+TwYLsd1F0cu6YFqO3+ODVxG pibNNHAGrL0sY46CxJIwyOINUB1v9eEGfkhwnmlZdtZf8MN7klpX9QhBp3NxYMuzA4+p bq6xsh4NNNfB9PG+HdHcACyPJwHfAXVZgAn5QM7kUD9B26NIC6fgmVNfspuJdpa7YsDE o9y+DjHNsFgc2NKUlU1p8u1azJNikuDnr8hvC5XgdAZ9WFEENA1zVSL1LSkpHUMP9aw6 DtEw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=TlcSHO0PYUHPy0BBchFGR15kwnEcxtL9LEFFMBual28=; b=BnKP5VzstLHiiEBsQ7k00VWRWgoYGMEuQ1sC5XJP3OQB88y8A0kQ7LfctYp1ZG1QwP PyfRDYJyhE3A5Y12PCzBc2p3aMFMI1SSQNiG3moTKWarB2aqVQYgp5iKoNRDNn8z7VZx Rzp5p+SVaXMJjrj2SqQO0OT+9nN5i3cJ/tlkaYFr6SfZ+ILaWBC3g6bH19H90wSgB07a M8ZyfShZWc74/M+2W0ajW3vpu1mMp79AeC/brurprSMyqcfh8obBuBoSFkeFBYGIEFG3 m3LULNn1MWXh8H5T97RvMZTMpa0ZkcuYbJI4C5vvfTMF5EBfOaPwpoJtJ6uB02tptFi6 iWEQ== X-Gm-Message-State: AOAM532HxaZ38WOlGqd55HQ626HZznGgzimLjnfNnbrpzyr2ortDXGIr uzhc+e6/5aFx0jQ4CSpRYy77NJgEdnGW2VaCvi57/A== X-Google-Smtp-Source: ABdhPJwpXcuYarkmjiJhRAAg69pJxyIO9GBdLu+aFvZuQm93bfvmmOrnXCAd7/QrLJ/o0nlKzOzbB1H1NdEUmcMJsBM= X-Received: by 2002:a05:6e02:c6b:: with SMTP id f11mr265563ilj.140.1623252777680; Wed, 09 Jun 2021 08:32:57 -0700 (PDT) MIME-Version: 1.0 References: <901ae36f-c5f1-4b0d-af17-90d29c212f78@Spark> In-Reply-To: <901ae36f-c5f1-4b0d-af17-90d29c212f78@Spark> From: Nathan Owens Date: Wed, 9 Jun 2021 08:32:46 -0700 Message-ID: To: Mike Puchol Cc: Dave Taht , starlink@lists.bufferbloat.net Content-Type: multipart/alternative; boundary="00000000000031d7bb05c456fdf8" Subject: Re: [Starlink] dynamically adjusting cake to starlink X-BeenThere: starlink@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Starlink has bufferbloat. Bad." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Jun 2021 15:32:59 -0000 --00000000000031d7bb05c456fdf8 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable > At any given time, a satellite may have ~10k cells under its footprint that it can serve, and GSO exclusions can exclude between 15% at high latitudes and 25% near the Equator. It has three arrays for downlinks and one array for uplinks, with 16 beams per array. Thus, it can train up to 48 beams towards the ground, and receive from up to 16 locations on the ground at any given time. Thus, to cover each cell, there is a physical TDM element, plus an additional multiplexing element for each terminal in each cell. If we assume a satellite has a capacity of ~20 Gbps, limited by the Ka links from satellite to gateway, each spot beam is capable of delivering ~416 Mbps through each spot beam. If we assume symmetrical performance, the 16 uplink beams from the one panel would =E2=80=9Ctake in=E2=80=9D ~416 Mbp= s per beam. I think based on the gateway docs from Anatel, we know this number is probably closer to ~13-16Gbps per Ka antenna, with likely only 1 active. Given not every user would be receiving data equally, the peak speed per cell is probably >400Mbps, we've seen speedtests over 500Mbps. I'm also guessing radio resource planning is pretty dynamic, or will need to be, I doubt it's a fixed 200/70ms kinda thing. On Wed, Jun 9, 2021 at 5:17 AM Mike Puchol wrote: > Greetings Dave, > > I have just been introduced into the world of bufferbloat - something I= =E2=80=99ve > had to deal with in my WISP, but never had a proper name for. I=E2=80=99m= an > aerospace engineer by origin, so please forgive my quite likely blunders = in > this space! > > As for the topic at hand, I understand the concept - in essence, if we ca= n > find what the Dishy - satellite - gateway triumvirate is doing in terms o= f > capacity and buffering, we can apply that information into our own router= =E2=80=99s > buffer/queue strategy. I have taken a quick look at the document, and the= re > is one assumption that is wrong, AFAIK, which is that the satellites are > =E2=80=9Cbent pipes=E2=80=9D - they actually process packets, which means= they are an > active component of buffering and queuing in the path (for optical links = to > work, satellites would necessarily have to process packets). > > At any given time, a satellite may have ~10k cells under its footprint > that it can serve, and GSO exclusions can exclude between 15% at high > latitudes and 25% near the Equator. It has three arrays for downlinks and > one array for uplinks, with 16 beams per array. Thus, it can train up to = 48 > beams towards the ground, and receive from up to 16 locations on the grou= nd > at any given time. Thus, to cover each cell, there is a physical TDM > element, plus an additional multiplexing element for each terminal in eac= h > cell. If we assume a satellite has a capacity of ~20 Gbps, limited by the > Ka links from satellite to gateway, each spot beam is capable of deliveri= ng > ~416 Mbps through each spot beam. If we assume symmetrical performance, t= he > 16 uplink beams from the one panel would =E2=80=9Ctake in=E2=80=9D ~416 M= bps per beam. > > Thus, a satellite is effectively limited to a 75/25 duty cycle between > downlink and uplink, as it can paint each cell three times for download, > for every time it receives from it. > > We then get to the killer - how many cells can a satellite effectively > serve? If we take 8000 cells as an average, each downlink spot beam would > need to care for ~167 cells, and in uplink ~500 cells. Focusing on uplink= , > each cell gets ~2ms of =E2=80=9Cbeam time=E2=80=9D to transmit, which is = ridiculously low. > From my tracker simulations, certain areas are served by around 8 > satellites at any given time, so catering for satellite spacing, we could > assume 10ms of =E2=80=9Cbeam time=E2=80=9D. Still to low, as this would o= nly allow for ~170 > packets. > > My current estimate from back of the envelope calculations is that > Starlink uses cell TDM factors per beam of 1 to 5, depending on served > customer density under the footprint. Thus, worst case, a satellite is > giving a cell ~200ms of downlink time and ~67ms of uplink time. A Dishy > terminal will need to buffer traffic while it gets its turn on a spot bea= m. > > Does this make sense, also from observations? > > Best, > > Mike > On Jun 9, 2021, 11:12 +0200, Dave Taht , wrote: > > Dear Mike: > > The biggest thing we need is something that can pull the right stats off > the dishy and dynamically adjust cake (on outbound at least) to have the > right amount of buffering for the available bandwidth, so as to make for > better statistical multiplexing (FQ) and active queue management (AQM) > > It=E2=80=99s pretty simple: in mangled shell script syntax: > > while up, down =3D getstats() > do > tc qdisc change dev eth0 root cake bandwidth $up > tc qdisc change dev ifb0 root cake bandwidth $down > done > > Which any router directly in front of the dishy can do (which is what > we=E2=80=99ve been doing) > > But whatever magic =E2=80=9Cgetstats()=E2=80=9D would need to do is uncle= ar from the stats > we get out of it, and a better alternative would be for the dishy itself > and their headends to be doing this with =E2=80=9CBQL" backpressure. > > As for the huge reductions of latency and jitter under working load, and = a > vast improvement in QoE - for what we=E2=80=99ve been able to achieve thu= s far, see > appendix A here: > > > https://docs.google.com/document/d/1rVGC-iNq2NZ0jk4f3IAiVUHz2S9O6P-F3vVZU= 2yBYtw/edit?usp=3Dsharing > > We=E2=80=99ve got plenty more data > on uploads and downloads and other forms of traffic (starlink is > optimizing for ping, only, over ipv6. Sigh)=E2=80=A6 > > =E2=80=A6 and a meeting with some starlink execs at 11AM today. > > I=E2=80=99m pretty sure at this point we will be able to make a massive > improvement in starlink=E2=80=99s network design very quickly, after that= meeting. > > On Jun 8, 2021, at 2:54 PM, Nathan Owens wrote: > > I invited Mike, the creator of the site (starlink.sx) to join the list - > he=E2=80=99s put a crazy amount of work in to figure out which sats are a= ctive > (with advice from Jonathan McDowell), programming GSO exclusion bands, et= c. > His dayjob is in the ISP business. > > On Sat, Jun 5, 2021 at 8:31 PM Darrell Budic > wrote: > >> https://starlink.sx if you have=E2=80=99t seen it yet. You can locate yo= urself, >> and it will make some educated guesses about which satellite to which >> ground station you=E2=80=99re using. Interesting to see the birds change= and the >> links move between ground stations, lots going on to make these things w= ork. >> _______________________________________________ >> Starlink mailing list >> Starlink@lists.bufferbloat.net >> https://lists.bufferbloat.net/listinfo/starlink >> > _______________________________________________ > Starlink mailing list > Starlink@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/starlink > > > _______________________________________________ > Starlink mailing list > Starlink@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/starlink > > --00000000000031d7bb05c456fdf8 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
>=C2=A0 At any given time, a satellite may have ~1= 0k cells under its footprint=20 that it can serve, and GSO exclusions can exclude between 15% at high=20 latitudes and 25% near the Equator. It has three arrays for downlinks=20 and one array for uplinks, with 16 beams per array. Thus, it can train=20 up to 48 beams towards the ground, and receive from up to 16 locations=20 on the ground at any given time. Thus, to cover each cell, there is a=20 physical TDM element, plus an additional multiplexing element for each=20 terminal in each cell. If we assume a satellite has a capacity of ~20=20 Gbps, limited by the Ka links from satellite to gateway, each spot beam=20 is capable of delivering ~416 Mbps through each spot beam. If we assume=20 symmetrical performance, the 16 uplink beams from the one panel would=20 =E2=80=9Ctake in=E2=80=9D ~416 Mbps per beam.

I think bas= ed on the gateway docs from Anatel, we know this number is probably closer = to ~13-16Gbps per Ka antenna, with likely only 1 active. Given not every us= er would be receiving data equally, the peak speed per cell is probably >= ;400Mbps, we've seen speedtests over 500Mbps.=C2=A0
I= 9;m also guessing radio resource planning is pretty dynamic, or will need t= o be, I doubt it's a fixed 200/70ms kinda thing.

On Wed, = Jun 9, 2021 at 5:17 AM Mike Puchol <mike@starlink.sx> wrote:
Greetings Dave,

I have just been introduced into the world of bufferbloat - something I=E2= =80=99ve had to deal with in my WISP, but never had a proper name for. I=E2= =80=99m an aerospace engineer by origin, so please forgive my quite likely = blunders in this space!

As for the topic at hand, I understand the concept - in essence, if we can = find what the Dishy - satellite - gateway triumvirate is doing in terms of = capacity and buffering, we can apply that information into our own router= =E2=80=99s buffer/queue strategy. I have taken a quick look at the document= , and there is one assumption that is wrong, AFAIK, which is that the satel= lites are =E2=80=9Cbent pipes=E2=80=9D - they actually process packets, whi= ch means they are an active component of buffering and queuing in the path = (for optical links to work, satellites would necessarily have to process pa= ckets).

At any given time, a satellite may have ~10k cells under its footprint that= it can serve, and GSO exclusions can exclude between 15% at high latitudes= and 25% near the Equator. It has three arrays for downlinks and one array = for uplinks, with 16 beams per array. Thus, it can train up to 48 beams tow= ards the ground, and receive from up to 16 locations on the ground at any g= iven time. Thus, to cover each cell, there is a physical TDM element, plus = an additional multiplexing element for each terminal in each cell. If we as= sume a satellite has a capacity of ~20 Gbps, limited by the Ka links from s= atellite to gateway, each spot beam is capable of delivering ~416 Mbps thro= ugh each spot beam. If we assume symmetrical performance, the 16 uplink bea= ms from the one panel would =E2=80=9Ctake in=E2=80=9D ~416 Mbps per beam.
Thus, a satellite is effectively limited to a 75/25 duty cycle between down= link and uplink, as it can paint each cell three times for download, for ev= ery time it receives from it.

We then get to the killer - how many cells can a satellite effectively serv= e? If we take 8000 cells as an average, each downlink spot beam would need = to care for ~167 cells, and in uplink ~500 cells. Focusing on uplink, each = cell gets ~2ms of =E2=80=9Cbeam time=E2=80=9D to transmit, which is ridicul= ously low. From my tracker simulations, certain areas are served by around = 8 satellites at any given time, so catering for satellite spacing, we could= assume 10ms of =E2=80=9Cbeam time=E2=80=9D. Still to low, as this would on= ly allow for ~170 packets.

My current estimate from back of the envelope calculations is that Starlink= uses cell TDM factors per beam of 1 to 5, depending on served customer den= sity under the footprint. Thus, worst case, a satellite is giving a cell ~2= 00ms of downlink time and ~67ms of uplink time. A Dishy terminal will need = to buffer traffic while it gets its turn on a spot beam.

Does this make sense, also from observations?

Best,

Mike
On Jun 9, 2021, 11:12 +0200, Dave Taht &l= t;davet@teklibre.ne= t>, wrote:
Dear Mike:

The biggest thing we need is something that can pull the right stats o= ff the dishy and dynamically adjust cake (on outbound at least) to have the= right amount of buffering for the available bandwidth, so as to make for b= etter statistical multiplexing (FQ) and active queue management (AQM)

It=E2=80=99s pretty simple: in mangled shell script syntax:

while up, down =3D getstats()
do
tc qdisc change dev eth0 root cake bandwidth $up
tc qdisc change dev ifb0 root cake bandwidth $down
done
=C2=A0
Which any router directly in front of the dishy can do (which is what = we=E2=80=99ve been doing)

But whatever magic =E2=80=9Cgetstats()=E2=80=9D would need to do is un= clear from the stats we get out of it, and a better alternative would be fo= r the dishy itself and their headends to be doing this with =E2=80=9CBQL&qu= ot; backpressure.

As for the huge reductions of latency and jitter under working load, a= nd a vast improvement in QoE - for what we=E2=80=99ve been able to achieve = thus far, see appendix A here:


We=E2=80=99ve got plenty more data=C2=A0
on uploads and downloads and other forms of traffic (starlink is optim= izing for ping, only, over ipv6. Sigh)=E2=80=A6

=E2=80=A6 and a meeting with some starlink execs at 11AM today.=C2=A0<= br>

I=E2=80=99m pretty sure at this point we will be able to make a massiv= e improvement in starlink=E2=80=99s network design very quickly, after that= meeting.=C2=A0

On Jun 8, 2021, at 2:54 PM, Nathan Owens <nathan@nathan.io> wrote:

I invited Mike, the creator of the site (starlink.sx) to join the list - he=E2= =80=99s put a crazy amount of work in to figure out which sats are active (= with advice from Jonathan McDowell), programming GSO exclusion bands, etc. = His dayjob is in the ISP business.

On Sat, Jun 5, 2021 at 8:31 PM Darrel= l Budic <bud= ic@onholyground.com> wrote:
https://starlink.sx= =C2=A0if you have=E2=80=99t seen it yet. You can locate yourself, and i= t will make some educated guesses about which satellite to which ground sta= tion you=E2=80=99re using. Interesting to see the birds change and the link= s move between ground stations, lots going on to make these things work. _______________________________________________
Starlink mailing list
Starlin= k@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/starlink
_______________________________________________
Starlink mailing list
Starlin= k@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/starlink

_______________________________________________
Starlink mailing list
Starlin= k@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/starlink
--00000000000031d7bb05c456fdf8--