Re: [Starlink] Model-Based Insights on the Performance, Fairness, and Stability of BBR (Dave Taht)

Re: [Starlink] Model-Based Insights on the Performance, Fairness, and Stability of BBR (Dave Taht)

[David P. Reed]

[-- Attachment #1: Type: text/plain, Size: 10749 bytes --]


> Date: Tue, 23 Aug 2022 11:07:47 -0700

> From: Dave Taht <dave.taht@gmail.com>
> I keep hoping that a paper will appear that compares BBR against the
> behaviors of pie, codel, fq-codel, fq-pie, and cake, and/or used
> figures from an actual problematic layer 2, like wifi or starlink, and
> compared against a fluid model.
 
So do I! Geez - bufferbloat is common in the network, and queue bloat in WiFi and Starlink, and even in Arista Networks (buffering, we have more than anyone else in our switches!) datacenter networks. If BBR can eliminate that excess queueing delay that destroys expected latencies, which some say (I haven't been able to observe it), then it should be demonstrable in a good model that includes such queueing.
 
But instead, a "fluid" approximation, which has nothing to do with actual fractal traffic demand observable in the real Internet, is proposed.
 
I'm grounded in fluid dynamics - I know what a Reynolds number is, and why it matters. I know what turbulence looks like in a fluid network. 
 
The problem of latency and latency variance is not a "fluid modelable" problem. It just isn't. When 3 packet inflows share an outflow from a fluid mixer, how do you predict how long it takes for a molecule of flow 1 to pass through the mixer vs. a molecule of flow 3?
 
But *latency* is what matters in the Internet. Not throughput.
 
> 
> This paper (using a fluid model for bbrv1 an bbrv2), at least,
> includes the RED AQM.
> 
> Also, right there, in fig 1, is slow start doing damage, not
> successfully modeled.
> 
> Aside from those nits, the paper gets better from there.
> 
> https://arxiv.org/pdf/2208.10103.pdf
> 
> --
> FQ World Domination pending: https://blog.cerowrt.org/post/state_of_fq_codel/
> Dave Täht CEO, TekLibre, LLC
> 
> 
> ------------------------------
> 
> Message: 2
> Date: Tue, 23 Aug 2022 11:54:27 -0700
> From: Dave Taht <dave.taht@gmail.com>
> To: "David P. Reed" <dpreed@deepplum.com>
> Cc: starlink@lists.bufferbloat.net, Cake List
> <cake@lists.bufferbloat.net>, Mike Luby <luby@bitripple.com>
> Subject: Re: [Starlink] starlink terminal mod chip
> Message-ID:
> <CAA93jw6EV0-Ni+HSf4Sy_vktuwWvWRnmimi+NoLW75AmMmDf-Q@mail.gmail.com>
> Content-Type: text/plain; charset="UTF-8"
> 
> I apologize for not replying sooner, I was trying to take a vacation
> and ended up ploughing
> through my backlog of paper instead, and trying to take apart some
> interesting data from a real
> ISPs pre-wireless-n network.
> 
> On Tue, Aug 16, 2022 at 12:29 PM David P. Reed via Starlink
> <starlink@lists.bufferbloat.net> wrote:
> >
> >
> >
> > On Sunday, August 14, 2022 12:00pm, starlink-request@lists.bufferbloat.net
> said:
> >
> > > Date: Sat, 13 Aug 2022 09:41:54 -0700
> > > From: Dave Taht <dave.taht@gmail.com>
> > > To: Dave Taht via Starlink <starlink@lists.bufferbloat.net>
> > > Subject: [Starlink] starlink terminal mod chip
> > >
> > > It's been my hope that starlink would merely install cake on the
> > > uplink themselves... but it's finally possible to get in there and do
> > > the work ourselves:
> > >
> > > https://github.com/KULeuven-COSIC/Starlink-FI
> >
> >
> >
> > As a big fan of Cake (and a happy user of it),
> 
> thx! :)
> 
> > I think it's fair to say that Starlink's capacity sharing method does not
> have a stable rate, almost certainly.
> 
> At which points in the network is that true? I am really vague on how
> they might be encoding symbols in general, and what sort of on/off
> periods are native to the mac? I have a deep "gut" feel only for how
> wifi behaves, and thought that they would be multiplexing terminals
> via bursting data over a short period from each. I don't really see
> that but the max resolution of the irtt data I have is 3ms. Shooting
> for 100us next.
> 
> There does not appear to be wifi-like retransmits, either, but I can
> hardly imagine mobility working without that.
> 
> Do they encode at at different rate at different distance?
> 
> > Has starlink published its link-scheduling approach?
> 
> Aside from the known 15s interval for potentially rerouting and/or
> changing bandwidth, no. Rigorously exploring the analog signalling
> up/down relative to attempts to use bandwidth in various ways
> may lend insight. An example might be after determining the exact 15s
> interval, a syn synack, and then waiting that interval before
> emitting an IW256 burst.
> 
> It often seems very much a network designed for speedtest rather than
> real traffic.
> 
> > Probably it is closer to polled (controlled by the ground station, not the
> terminal).
> 
> There are at least 3 tiers of service on starlink. I don't know if
> anyone has pulled the signals off when idle, semi-idle or otherwise.
> 
> > Cake assumes that the uplink to the public Internet is, at the link layer,
> stable and unvarying in its rate, so cake's control theory assumes that. The
> downlink assumption is similar.
> 
> This is overbroad. Cake's shaper assumes that. The codel component,
> without that engaged, feeding packets in via local backpressure, even
> over fairly large intervals, is fluid. The queue delay target needs to
> be greater than the transmit interval.
> 
> >
> >
> > So here's why I would NOT recommend just adding Cake to the starlink terminal
> - it's the wrong link model.
> 
> Heh. My hope was for link backpressure, the hw asking the qdisc to
> release packets to be encrypted an transmitted sufficiently enough
> usec before the txop arrived.
> 
> > Let's assume there is just one satellite that the terminal is using for its
> packets. The satellite is dealing with dozens of terminals at once. The
> satellite-to-ground-station link is more like a standard fixed rate link, being
> multiplexed among use by dozens of terminals.
> >
> >
> >
> > This terminal traffic is NOT the way a "home router" uses a cable modem or
> DSL or fiber modem.
> >
> >
> >
> > If you want to get something approaching minimal latency and fair-queueing
> (which isn't "fair" because the term "fair" just means starvation free in most
> queueing theory and networking applications), you need to drop packets or send
> congestion signals that are based on the *achievable rate* of the link.
> >
> >
> >
> > However, the achievable rate of the link is being dynamically managed by the
> *ground station* needing to balance traffic among all terminals generating flows
> through the ground station end.
> >
> >
> >
> > Most likely this will result in Cake and the ground station getting in a
> "battle" to achieve good utilization.
> 
> yes.
> 
> >
> >
> > The same issue arises if you try to use Cake in each node sharing a WiFi LAN.
> The WiFi protocols (including the polled mode in 802.11ax, but also the older
> 802.11 CSMA-CA Aloha style before 11ax) actively "schedule" the shared WLAN air
> link - either centrally or in a decentralized manner. Putting Cake as a congestion
> mechanism on a shared air link is problematic because the competing traffic causes
> the available capacity to vary tremendously, over short time intervals.
> 
> yes. I got some interesting data on that exact scenario (using a
> libreqos middlebox) which I'm still thinking about.
> 
> >
> >
> >
> > The problem here is that the timescale of the scheduling of link bandwidth is
> incompatible with the end-to-end TCP congestion control loops. This guarantees
> control theory instability.
> 
> While I don't disagree a cite or three would help me. Yes! unstable!
> How unstable?
> 
> My mental model, originally, was that the terminal scheduling was very
> simple TDMA... which is stable,
> predictable, and wasteful.
> 
> 
> >
> >
> > Now if Starlink had some good, smart people who understand Internet
> congestion control and queueing theory and control theory in practice (and some of
> those are participating on this list, probably easy to hire [hint]) ---
> 
> I hope they have people on the problem.
> 
> > the obvious place to incorporate Cake or FQ-codel in the path from terminal
> to public Internet would be *in the ground station* (or maybe in the satellite, if
> the satellite actually controls the scheduling of access).
> 
> This does require an oracle to some extent. I think that the scheduler
> does need to be on the ground stations, but fq+aqm belongs everywhere.
> 
> >
> >
> > But before we get thrilled about reverse-engineering the Starlink terminal,
> remember, Starlink's technology is quite different from a cable data network or a
> GPON network in its packet architecture. Simple replacement of its software or
> simple patches probably won't work.
> 
> It's just two easily dissassembled binaries, one for tx, the other for rx.
> 
> I have (delusionally) fond memories of porting the decompiled "empire"
> game around back in the 80s. Not sure how much better the tools have
> got (and this activity is expressly prohibited by the starlink eula)
> 
> >
> >
> > That said, the talk about breaking into the terminal is quite fascinating. I
> do believe Starlink has real defense-in-depth, though, so it's unlikely that
> Starlink other users are at all Pwned if you do that. Yeah, if its your terminal,
> you can probably sniff your traffic and mess up your use. It might be that you can
> disrupt (DoS) a satellite by "jamming" its "bent-pipe channels". But that's
> normal, nothing out of the ordinary. They will come and get you, and it is likely
> gonna get you jailed for a long time.
> 
> Their list of prohibitions and rules for bug bounty security
> researchers is reasonable.
> 
> The gateway for getting in there to actually improve the system is
> seemingly insurmountable.
> 
> >
> >
> >
> >
> > >
> > > (great blackhat talk, btw)
> 
> The one on john deere was also great
> 
> https://doctorow.medium.com/this-weekend-i-watched-a-hacker-jailbreak-a-john-deere-tractor-live-on-stage-febbb0dc5a76
> 
> >
> >
> > _______________________________________________
> > Starlink mailing list
> > Starlink@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/starlink
> 
> 
> 
> --
> FQ World Domination pending: https://blog.cerowrt.org/post/state_of_fq_codel/
> Dave Täht CEO, TekLibre, LLC
> 
> 
> ------------------------------
> 
> Subject: Digest Footer
> 
> _______________________________________________
> Starlink mailing list
> Starlink@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/starlink
> 
> 
> ------------------------------
> 
> End of Starlink Digest, Vol 17, Issue 16
> ****************************************
> 

[-- Attachment #2: Type: text/html, Size: 14335 bytes --]