<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">This was a fun talk. When I gave it I thought… "If I could only get *one* MAJOR ISP to update their routers and headends…"<div class=""><br class=""></div><div class=""><a href="https://blog.apnic.net/2020/01/22/bufferbloat-may-be-solved-but-its-not-over-yet/" class="">https://blog.apnic.net/2020/01/22/bufferbloat-may-be-solved-but-its-not-over-yet/</a></div><div class=""><br class=""></div><div class="">More below:<br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Jun 9, 2021, at 5:16 AM, Mike Puchol <<a href="mailto:mike@starlink.sx" class="">mike@starlink.sx</a>> wrote:</div><br class="Apple-interchange-newline"><div class="">
<title class=""></title>
<div class="">
<div name="messageBodySection" class="">
<div dir="auto" class="">Greetings Dave,<br class="">
<br class="">
I have just been introduced into the world of bufferbloat - something I’ve had to deal with in my WISP, but never had a proper name for. I’m an aerospace engineer by origin, so please forgive my quite likely blunders in this space!<br class="">
<br class=""></div></div></div></div></blockquote><div><br class=""></div>Not just you, by a long shot.</div><div><br class=""><blockquote type="cite" class=""><div class=""><div class=""><div name="messageBodySection" class=""><div dir="auto" class="">
As for the topic at hand, I understand the concept - in essence, if we can find what the Dishy - satellite - gateway triumvirate is doing in terms of capacity and buffering, we can apply that information into our own router’s buffer/queue strategy. I have taken a quick look at the document, and there is one assumption that is wrong, AFAIK, which is that the satellites are “bent pipes” - they actually process packets, which means they are an active component of buffering and queuing in the path (for optical links to work, satellites would necessarily have to process packets).<br class=""></div></div></div></div></blockquote><div><br class=""></div>Both my other satcom experts were making the assumption it was actually a bent pipe at this point. I had thought it did packets, simply because I didn’t</div><div>grok how they could possibly globally optimize the world solution any other way. </div><div><br class=""><blockquote type="cite" class=""><div class=""><div class=""><div name="messageBodySection" class=""><div dir="auto" class="">
<br class="">
At any given time, a satellite may have ~10k cells under its footprint that it can serve, and GSO exclusions can exclude between 15% at high latitudes and 25% near the Equator. It has three arrays for downlinks and one array for uplinks, with 16 beams per array. Thus, it can train up to 48 beams towards the ground, and receive from up to 16 locations on the ground at any given time. Thus, to cover each cell, there is a physical TDM element, plus an additional multiplexing element for each terminal in each cell. If we assume a satellite has a capacity of ~20 Gbps, limited by the Ka links from satellite to gateway, each spot beam is capable of delivering ~416 Mbps through each spot beam. If we assume symmetrical performance, the 16 uplink beams from the one panel would “take in” ~416 Mbps per beam.<br class="">
<br class="">
Thus, a satellite is effectively limited to a 75/25 duty cycle between downlink and uplink, as it can paint each cell three times for download, for every time it receives from it.<br class="">
<br class="">
We then get to the killer - how many cells can a satellite effectively serve? If we take 8000 cells as an average, each downlink spot beam would need to care for ~167 cells, and in uplink ~500 cells. Focusing on uplink, each cell gets ~2ms of “beam time” to transmit, which is ridiculously low. From my tracker simulations, certain areas are served by around 8 satellites at any given time, so catering for satellite spacing, we could assume 10ms of “beam time”. Still to low, as this would only allow for ~170 packets.<br class="">
<br class="">
My current estimate from back of the envelope calculations is that Starlink uses cell TDM factors per beam of 1 to 5, depending on served customer density under the footprint. Thus, worst case, a satellite is giving a cell ~200ms of downlink time and ~67ms of uplink time. A Dishy terminal will need to buffer traffic while it gets its turn on a spot beam.<br class="">
<br class="">
Does this make sense, also from observations?<br class=""></div>
</div>
<div name="messageSignatureSection" class=""><br class="">
<div class="matchFont">Best,<br class="">
<br class="">
Mike</div>
</div>
<div name="messageReplySection" class="">On Jun 9, 2021, 11:12 +0200, Dave Taht <<a href="mailto:davet@teklibre.net" class="">davet@teklibre.net</a>>, wrote:<br class="">
<blockquote type="cite" style="border-left-color: grey; border-left-width: thin; border-left-style: solid; margin: 5px 5px;padding-left: 10px;" class="">Dear Mike:
<div class=""><br class=""></div>
<div class="">The biggest thing we need is something that can pull the right stats off the dishy and dynamically adjust cake (on outbound at least) to have the right amount of buffering for the available bandwidth, so as to make for better statistical multiplexing (FQ) and active queue management (AQM)
<div class=""><br class=""></div>
<div class="">It’s pretty simple: in mangled shell script syntax:</div>
<div class=""><br class=""></div>
<div class="">while up, down = getstats()</div>
<div class="">do</div>
<div class="">tc qdisc change dev eth0 root cake bandwidth $up</div>
<div class="">tc qdisc change dev ifb0 root cake bandwidth $down</div>
<div class="">done</div>
<div class=""> </div>
<div class="">Which any router directly in front of the dishy can do (which is what we’ve been doing)</div>
<div class=""><br class=""></div>
<div class="">But whatever magic “getstats()” would need to do is unclear from the stats we get out of it, and a better alternative would be for the dishy itself and their headends to be doing this with “BQL" backpressure.</div>
<div class=""><br class=""></div>
<div class="">As for the huge reductions of latency and jitter under working load, and a vast improvement in QoE - for what we’ve been able to achieve thus far, see appendix A here:</div>
<div class=""><br class=""></div>
<div class=""><a href="https://docs.google.com/document/d/1rVGC-iNq2NZ0jk4f3IAiVUHz2S9O6P-F3vVZU2yBYtw/edit?usp=sharing" class="">https://docs.google.com/document/d/1rVGC-iNq2NZ0jk4f3IAiVUHz2S9O6P-F3vVZU2yBYtw/edit?usp=sharing</a></div>
<div class=""><br class=""></div>
<div class="">We’ve got plenty more data </div>
<div class="">on uploads and downloads and other forms of traffic (starlink is optimizing for ping, only, over ipv6. Sigh)…</div>
<div class=""><br class=""></div>
<div class="">… and a meeting with some starlink execs at 11AM today. <br class="">
<div class=""><br class=""></div>
<div class="">I’m pretty sure at this point we will be able to make a massive improvement in starlink’s network design very quickly, after that meeting. <br class="">
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On Jun 8, 2021, at 2:54 PM, Nathan Owens <<a href="mailto:nathan@nathan.io" class="">nathan@nathan.io</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div dir="ltr" class="">
<div dir="auto" class="">I invited Mike, the creator of the site (<a href="http://starlink.sx/" class="">starlink.sx</a>) to join the list - he’s put a crazy amount of work in to figure out which sats are active (with advice from Jonathan McDowell), programming GSO exclusion bands, etc. His dayjob is in the ISP business.<br class=""></div>
</div>
<div class=""><br class="">
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Sat, Jun 5, 2021 at 8:31 PM Darrell Budic <<a href="mailto:budic@onholyground.com" target="_blank" class="">budic@onholyground.com</a>> wrote:<br class=""></div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div style="overflow-wrap: break-word;" class=""><a href="https://starlink.sx/" target="_blank" class="">https://starlink.sx</a> if you have’t seen it yet. You can locate yourself, and it will make some educated guesses about which satellite to which ground station you’re using. Interesting to see the birds change and the links move between ground stations, lots going on to make these things work.</div>
_______________________________________________<br class="">
Starlink mailing list<br class="">
<a href="mailto:Starlink@lists.bufferbloat.net" target="_blank" class="">Starlink@lists.bufferbloat.net</a><br class="">
<a href="https://lists.bufferbloat.net/listinfo/starlink" rel="noreferrer" target="_blank" class="">https://lists.bufferbloat.net/listinfo/starlink</a><br class=""></blockquote>
</div>
</div>
_______________________________________________<br class="">
Starlink mailing list<br class="">
<a href="mailto:Starlink@lists.bufferbloat.net" class="">Starlink@lists.bufferbloat.net</a><br class="">
<a href="https://lists.bufferbloat.net/listinfo/starlink" class="">https://lists.bufferbloat.net/listinfo/starlink</a><br class=""></div>
</blockquote>
</div>
<br class=""></div>
</div>
</div>
_______________________________________________<br class="">
Starlink mailing list<br class="">
<a href="mailto:Starlink@lists.bufferbloat.net" class="">Starlink@lists.bufferbloat.net</a><br class="">
<a href="https://lists.bufferbloat.net/listinfo/starlink" class="">https://lists.bufferbloat.net/listinfo/starlink</a><br class=""></blockquote>
</div>
</div>
</div></blockquote></div><br class=""></div></body></html>