From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp84.iad3a.emailsrvr.com (smtp84.iad3a.emailsrvr.com [173.203.187.84]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 6C6233CB37 for ; Wed, 24 Aug 2022 14:25:08 -0400 (EDT) Received: from app51.wa-webapps.iad3a (relay-webapps.rsapps.net [172.27.255.140]) by smtp3.relay.iad3a.emailsrvr.com (SMTP Server) with ESMTP id D4D49239E1 for ; Wed, 24 Aug 2022 14:25:07 -0400 (EDT) Received: from deepplum.com (localhost.localdomain [127.0.0.1]) by app51.wa-webapps.iad3a (Postfix) with ESMTP id C0C54A00C1 for ; Wed, 24 Aug 2022 14:25:07 -0400 (EDT) Received: by apps.rackspace.com (Authenticated sender: dpreed@deepplum.com, from: dpreed@deepplum.com) with HTTP; Wed, 24 Aug 2022 14:25:07 -0400 (EDT) X-Auth-ID: dpreed@deepplum.com Date: Wed, 24 Aug 2022 14:25:07 -0400 (EDT) From: "David P. Reed" To: starlink@lists.bufferbloat.net MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_20220824142507000000_29345" Importance: Normal X-Priority: 3 (Normal) X-Type: html In-Reply-To: References: X-Client-IP: 209.6.168.128 Message-ID: <1661365507.78588851@apps.rackspace.com> X-Mailer: webmail/19.0.18-RC X-Classification-ID: 595f1a78-538b-458c-a229-f7a999e415cd-1-1 Subject: Re: [Starlink] =?utf-8?q?Model-Based_Insights_on_the_Performance=2C_F?= =?utf-8?q?airness=2C_and_Stability_of_BBR_=28Dave_Taht=29?= X-BeenThere: starlink@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Starlink has bufferbloat. Bad." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Aug 2022 18:25:08 -0000 ------=_20220824142507000000_29345 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable =0A> Date: Tue, 23 Aug 2022 11:07:47 -0700=0A=0A> From: Dave Taht =0A> I keep hoping that a paper will appear that compares BBR a= gainst the=0A> behaviors of pie, codel, fq-codel, fq-pie, and cake, and/or = used=0A> figures from an actual problematic layer 2, like wifi or starlink,= and=0A> compared against a fluid model.=0A =0ASo do I! Geez - bufferbloat = is common in the network, and queue bloat in WiFi and Starlink, and even in= Arista Networks (buffering, we have more than anyone else in our switches!= ) datacenter networks. If BBR can eliminate that excess queueing delay that= destroys expected latencies, which some say (I haven't been able to observ= e it), then it should be demonstrable in a good model that includes such qu= eueing.=0A =0ABut instead, a "fluid" approximation, which has nothing to do= with actual fractal traffic demand observable in the real Internet, is pro= posed.=0A =0AI'm grounded in fluid dynamics - I know what a Reynolds number= is, and why it matters. I know what turbulence looks like in a fluid netwo= rk. =0A =0AThe problem of latency and latency variance is not a "fluid mode= lable" problem. It just isn't. When 3 packet inflows share an outflow from = a fluid mixer, how do you predict how long it takes for a molecule of flow = 1 to pass through the mixer vs. a molecule of flow 3?=0A =0ABut *latency* i= s what matters in the Internet. Not throughput.=0A =0A> =0A> This paper (us= ing a fluid model for bbrv1 an bbrv2), at least,=0A> includes the RED AQM.= =0A> =0A> Also, right there, in fig 1, is slow start doing damage, not=0A> = successfully modeled.=0A> =0A> Aside from those nits, the paper gets better= from there.=0A> =0A> https://arxiv.org/pdf/2208.10103.pdf=0A> =0A> --=0A> = FQ World Domination pending: https://blog.cerowrt.org/post/state_of_fq_code= l/=0A> Dave T=C3=A4ht CEO, TekLibre, LLC=0A> =0A> =0A> --------------------= ----------=0A> =0A> Message: 2=0A> Date: Tue, 23 Aug 2022 11:54:27 -0700=0A= > From: Dave Taht =0A> To: "David P. Reed" =0A> Cc: starlink@lists.bufferbloat.net, Cake List=0A> , Mike Luby =0A> Subject: Re: [Starli= nk] starlink terminal mod chip=0A> Message-ID:=0A> =0A> Content-Type: text/plain; = charset=3D"UTF-8"=0A> =0A> I apologize for not replying sooner, I was tryin= g to take a vacation=0A> and ended up ploughing=0A> through my backlog of p= aper instead, and trying to take apart some=0A> interesting data from a rea= l=0A> ISPs pre-wireless-n network.=0A> =0A> On Tue, Aug 16, 2022 at 12:29 P= M David P. Reed via Starlink=0A> wrote:=0A= > >=0A> >=0A> >=0A> > On Sunday, August 14, 2022 12:00pm, starlink-request@= lists.bufferbloat.net=0A> said:=0A> >=0A> > > Date: Sat, 13 Aug 2022 09:41:= 54 -0700=0A> > > From: Dave Taht =0A> > > To: Dave Tah= t via Starlink =0A> > > Subject: [Starlink]= starlink terminal mod chip=0A> > >=0A> > > It's been my hope that starlink= would merely install cake on the=0A> > > uplink themselves... but it's fin= ally possible to get in there and do=0A> > > the work ourselves:=0A> > >=0A= > > > https://github.com/KULeuven-COSIC/Starlink-FI=0A> >=0A> >=0A> >=0A> >= As a big fan of Cake (and a happy user of it),=0A> =0A> thx! :)=0A> =0A> >= I think it's fair to say that Starlink's capacity sharing method does not= =0A> have a stable rate, almost certainly.=0A> =0A> At which points in the = network is that true? I am really vague on how=0A> they might be encoding s= ymbols in general, and what sort of on/off=0A> periods are native to the ma= c? I have a deep "gut" feel only for how=0A> wifi behaves, and thought that= they would be multiplexing terminals=0A> via bursting data over a short pe= riod from each. I don't really see=0A> that but the max resolution of the i= rtt data I have is 3ms. Shooting=0A> for 100us next.=0A> =0A> There does no= t appear to be wifi-like retransmits, either, but I can=0A> hardly imagine = mobility working without that.=0A> =0A> Do they encode at at different rate= at different distance?=0A> =0A> > Has starlink published its link-scheduli= ng approach?=0A> =0A> Aside from the known 15s interval for potentially rer= outing and/or=0A> changing bandwidth, no. Rigorously exploring the analog s= ignalling=0A> up/down relative to attempts to use bandwidth in various ways= =0A> may lend insight. An example might be after determining the exact 15s= =0A> interval, a syn synack, and then waiting that interval before=0A> emit= ting an IW256 burst.=0A> =0A> It often seems very much a network designed f= or speedtest rather than=0A> real traffic.=0A> =0A> > Probably it is closer= to polled (controlled by the ground station, not the=0A> terminal).=0A> = =0A> There are at least 3 tiers of service on starlink. I don't know if=0A>= anyone has pulled the signals off when idle, semi-idle or otherwise.=0A> = =0A> > Cake assumes that the uplink to the public Internet is, at the link = layer,=0A> stable and unvarying in its rate, so cake's control theory assum= es that. The=0A> downlink assumption is similar.=0A> =0A> This is overbroad= . Cake's shaper assumes that. The codel component,=0A> without that engaged= , feeding packets in via local backpressure, even=0A> over fairly large int= ervals, is fluid. The queue delay target needs to=0A> be greater than the t= ransmit interval.=0A> =0A> >=0A> >=0A> > So here's why I would NOT recommen= d just adding Cake to the starlink terminal=0A> - it's the wrong link model= .=0A> =0A> Heh. My hope was for link backpressure, the hw asking the qdisc = to=0A> release packets to be encrypted an transmitted sufficiently enough= =0A> usec before the txop arrived.=0A> =0A> > Let's assume there is just on= e satellite that the terminal is using for its=0A> packets. The satellite i= s dealing with dozens of terminals at once. The=0A> satellite-to-ground-sta= tion link is more like a standard fixed rate link, being=0A> multiplexed am= ong use by dozens of terminals.=0A> >=0A> >=0A> >=0A> > This terminal traff= ic is NOT the way a "home router" uses a cable modem or=0A> DSL or fiber mo= dem.=0A> >=0A> >=0A> >=0A> > If you want to get something approaching minim= al latency and fair-queueing=0A> (which isn't "fair" because the term "fair= " just means starvation free in most=0A> queueing theory and networking app= lications), you need to drop packets or send=0A> congestion signals that ar= e based on the *achievable rate* of the link.=0A> >=0A> >=0A> >=0A> > Howev= er, the achievable rate of the link is being dynamically managed by the=0A>= *ground station* needing to balance traffic among all terminals generating= flows=0A> through the ground station end.=0A> >=0A> >=0A> >=0A> > Most lik= ely this will result in Cake and the ground station getting in a=0A> "battl= e" to achieve good utilization.=0A> =0A> yes.=0A> =0A> >=0A> >=0A> > The sa= me issue arises if you try to use Cake in each node sharing a WiFi LAN.=0A>= The WiFi protocols (including the polled mode in 802.11ax, but also the ol= der=0A> 802.11 CSMA-CA Aloha style before 11ax) actively "schedule" the sha= red WLAN air=0A> link - either centrally or in a decentralized manner. Putt= ing Cake as a congestion=0A> mechanism on a shared air link is problematic = because the competing traffic causes=0A> the available capacity to vary tre= mendously, over short time intervals.=0A> =0A> yes. I got some interesting = data on that exact scenario (using a=0A> libreqos middlebox) which I'm stil= l thinking about.=0A> =0A> >=0A> >=0A> >=0A> > The problem here is that the= timescale of the scheduling of link bandwidth is=0A> incompatible with the= end-to-end TCP congestion control loops. This guarantees=0A> control theor= y instability.=0A> =0A> While I don't disagree a cite or three would help m= e. Yes! unstable!=0A> How unstable?=0A> =0A> My mental model, originally, w= as that the terminal scheduling was very=0A> simple TDMA... which is stable= ,=0A> predictable, and wasteful.=0A> =0A> =0A> >=0A> >=0A> > Now if Starlin= k had some good, smart people who understand Internet=0A> congestion contro= l and queueing theory and control theory in practice (and some of=0A> those= are participating on this list, probably easy to hire [hint]) ---=0A> =0A>= I hope they have people on the problem.=0A> =0A> > the obvious place to in= corporate Cake or FQ-codel in the path from terminal=0A> to public Internet= would be *in the ground station* (or maybe in the satellite, if=0A> the sa= tellite actually controls the scheduling of access).=0A> =0A> This does req= uire an oracle to some extent. I think that the scheduler=0A> does need to = be on the ground stations, but fq+aqm belongs everywhere.=0A> =0A> >=0A> >= =0A> > But before we get thrilled about reverse-engineering the Starlink te= rminal,=0A> remember, Starlink's technology is quite different from a cable= data network or a=0A> GPON network in its packet architecture. Simple repl= acement of its software or=0A> simple patches probably won't work.=0A> =0A>= It's just two easily dissassembled binaries, one for tx, the other for rx.= =0A> =0A> I have (delusionally) fond memories of porting the decompiled "em= pire"=0A> game around back in the 80s. Not sure how much better the tools h= ave=0A> got (and this activity is expressly prohibited by the starlink eula= )=0A> =0A> >=0A> >=0A> > That said, the talk about breaking into the termin= al is quite fascinating. I=0A> do believe Starlink has real defense-in-dept= h, though, so it's unlikely that=0A> Starlink other users are at all Pwned = if you do that. Yeah, if its your terminal,=0A> you can probably sniff your= traffic and mess up your use. It might be that you can=0A> disrupt (DoS) a= satellite by "jamming" its "bent-pipe channels". But that's=0A> normal, no= thing out of the ordinary. They will come and get you, and it is likely=0A>= gonna get you jailed for a long time.=0A> =0A> Their list of prohibitions = and rules for bug bounty security=0A> researchers is reasonable.=0A> =0A> T= he gateway for getting in there to actually improve the system is=0A> seemi= ngly insurmountable.=0A> =0A> >=0A> >=0A> >=0A> >=0A> > >=0A> > > (great bl= ackhat talk, btw)=0A> =0A> The one on john deere was also great=0A> =0A> ht= tps://doctorow.medium.com/this-weekend-i-watched-a-hacker-jailbreak-a-john-= deere-tractor-live-on-stage-febbb0dc5a76=0A> =0A> >=0A> >=0A> > ___________= ____________________________________=0A> > Starlink mailing list=0A> > Star= link@lists.bufferbloat.net=0A> > https://lists.bufferbloat.net/listinfo/sta= rlink=0A> =0A> =0A> =0A> --=0A> FQ World Domination pending: https://blog.c= erowrt.org/post/state_of_fq_codel/=0A> Dave T=C3=A4ht CEO, TekLibre, LLC=0A= > =0A> =0A> ------------------------------=0A> =0A> Subject: Digest Footer= =0A> =0A> _______________________________________________=0A> Starlink mail= ing list=0A> Starlink@lists.bufferbloat.net=0A> https://lists.bufferbloat.n= et/listinfo/starlink=0A> =0A> =0A> ------------------------------=0A> =0A> = End of Starlink Digest, Vol 17, Issue 16=0A> ******************************= **********=0A> ------=_20220824142507000000_29345 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

> Date: Tue, 23 Aug= 2022 11:07:47 -0700

=0A
=0A

> From: Dave Taht <dave.taht@gmail.com>
> I keep = hoping that a paper will appear that compares BBR against the
> beh= aviors of pie, codel, fq-codel, fq-pie, and cake, and/or used
> fig= ures from an actual problematic layer 2, like wifi or starlink, and
&g= t; compared against a fluid model.

=0A

 

=0A=

So do I! Geez - bufferbloat is common in the network, = and queue bloat in WiFi and Starlink, and even in Arista Networks (bufferin= g, we have more than anyone else in our switches!) datacenter networks. If = BBR can eliminate that excess queueing delay that destroys expected latenci= es, which some say (I haven't been able to observe it), then it should be d= emonstrable in a good model that includes such queueing.

=0A

 

=0A

But instead, a "fluid" approxima= tion, which has nothing to do with actual fractal traffic demand observable= in the real Internet, is proposed.

=0A

 

= =0A

I'm grounded in fluid dynamics - I know what a Reyn= olds number is, and why it matters. I know what turbulence looks like in a = fluid network. 

=0A

 

=0A

The problem of latency and latency variance is not a "fluid modelabl= e" problem. It just isn't. When 3 packet inflows share an outflow from a fl= uid mixer, how do you predict how long it takes for a molecule of flow 1 to= pass through the mixer vs. a molecule of flow 3?

=0A

But *latency* is what matters in the In= ternet. Not throughput.

=0A

 

=0A

>
> This paper (using a fluid model for bbrv1 an bbrv= 2), at least,
> includes the RED AQM.
>
> Also, ri= ght there, in fig 1, is slow start doing damage, not
> successfully= modeled.
>
> Aside from those nits, the paper gets better= from there.
>
> https://arxiv.org/pdf/2208.10103.pdf
>
> --
> FQ World Domination pending: https://blog.cer= owrt.org/post/state_of_fq_codel/
> Dave T=C3=A4ht CEO, TekLibre, LL= C
>
>
> ------------------------------
> =
> Message: 2
> Date: Tue, 23 Aug 2022 11:54:27 -0700
= > From: Dave Taht <dave.taht@gmail.com>
> To: "David P. Re= ed" <dpreed@deepplum.com>
> Cc: starlink@lists.bufferbloat.ne= t, Cake List
> <cake@lists.bufferbloat.net>, Mike Luby <lu= by@bitripple.com>
> Subject: Re: [Starlink] starlink terminal mo= d chip
> Message-ID:
> <CAA93jw6EV0-Ni+HSf4Sy_vktuwWvWRn= mimi+NoLW75AmMmDf-Q@mail.gmail.com>
> Content-Type: text/plain; = charset=3D"UTF-8"
>
> I apologize for not replying sooner,= I was trying to take a vacation
> and ended up ploughing
>= through my backlog of paper instead, and trying to take apart some
&g= t; interesting data from a real
> ISPs pre-wireless-n network.
>
> On Tue, Aug 16, 2022 at 12:29 PM David P. Reed via Starlin= k
> <starlink@lists.bufferbloat.net> wrote:
> >> >
> >
> > On Sunday, August 14, 2022 12:00= pm, starlink-request@lists.bufferbloat.net
> said:
> >> > > Date: Sat, 13 Aug 2022 09:41:54 -0700
> > >= ; From: Dave Taht <dave.taht@gmail.com>
> > > To: Dave = Taht via Starlink <starlink@lists.bufferbloat.net>
> > >= ; Subject: [Starlink] starlink terminal mod chip
> > >
&= gt; > > It's been my hope that starlink would merely install cake on = the
> > > uplink themselves... but it's finally possible to g= et in there and do
> > > the work ourselves:
> > &= gt;
> > > https://github.com/KULeuven-COSIC/Starlink-FI
= > >
> >
> >
> > As a big fan of Cake= (and a happy user of it),
>
> thx! :)
>
>= ; > I think it's fair to say that Starlink's capacity sharing method doe= s not
> have a stable rate, almost certainly.
>
> = At which points in the network is that true? I am really vague on how
= > they might be encoding symbols in general, and what sort of on/off
> periods are native to the mac? I have a deep "gut" feel only for how=
> wifi behaves, and thought that they would be multiplexing termin= als
> via bursting data over a short period from each. I don't real= ly see
> that but the max resolution of the irtt data I have is 3ms= . Shooting
> for 100us next.
>
> There does not ap= pear to be wifi-like retransmits, either, but I can
> hardly imagin= e mobility working without that.
>
> Do they encode at at = different rate at different distance?
>
> > Has starlin= k published its link-scheduling approach?
>
> Aside from t= he known 15s interval for potentially rerouting and/or
> changing b= andwidth, no. Rigorously exploring the analog signalling
> up/down = relative to attempts to use bandwidth in various ways
> may lend in= sight. An example might be after determining the exact 15s
> interv= al, a syn synack, and then waiting that interval before
> emitting = an IW256 burst.
>
> It often seems very much a network des= igned for speedtest rather than
> real traffic.
>
>= ; > Probably it is closer to polled (controlled by the ground station, n= ot the
> terminal).
>
> There are at least 3 tiers= of service on starlink. I don't know if
> anyone has pulled the si= gnals off when idle, semi-idle or otherwise.
>
> > Cake= assumes that the uplink to the public Internet is, at the link layer,
> stable and unvarying in its rate, so cake's control theory assumes th= at. The
> downlink assumption is similar.
>
> This= is overbroad. Cake's shaper assumes that. The codel component,
> w= ithout that engaged, feeding packets in via local backpressure, even
&= gt; over fairly large intervals, is fluid. The queue delay target needs to<= br />> be greater than the transmit interval.
>
> ><= br />> >
> > So here's why I would NOT recommend just addi= ng Cake to the starlink terminal
> - it's the wrong link model.
>
> Heh. My hope was for link backpressure, the hw asking the= qdisc to
> release packets to be encrypted an transmitted sufficie= ntly enough
> usec before the txop arrived.
>
> &g= t; Let's assume there is just one satellite that the terminal is using for = its
> packets. The satellite is dealing with dozens of terminals at= once. The
> satellite-to-ground-station link is more like a standa= rd fixed rate link, being
> multiplexed among use by dozens of term= inals.
> >
> >
> >
> > This ter= minal traffic is NOT the way a "home router" uses a cable modem or
>= ; DSL or fiber modem.
> >
> >
> >
>= ; > If you want to get something approaching minimal latency and fair-qu= eueing
> (which isn't "fair" because the term "fair" just means sta= rvation free in most
> queueing theory and networking applications)= , you need to drop packets or send
> congestion signals that are ba= sed on the *achievable rate* of the link.
> >
> >
> >
> > However, the achievable rate of the link is bein= g dynamically managed by the
> *ground station* needing to balance = traffic among all terminals generating flows
> through the ground s= tation end.
> >
> >
> >
> > Mos= t likely this will result in Cake and the ground station getting in a
= > "battle" to achieve good utilization.
>
> yes.
&= gt;
> >
> >
> > The same issue arises if = you try to use Cake in each node sharing a WiFi LAN.
> The WiFi pro= tocols (including the polled mode in 802.11ax, but also the older
>= 802.11 CSMA-CA Aloha style before 11ax) actively "schedule" the shared WLA= N air
> link - either centrally or in a decentralized manner. Putti= ng Cake as a congestion
> mechanism on a shared air link is problem= atic because the competing traffic causes
> the available capacity = to vary tremendously, over short time intervals.
>
> yes. = I got some interesting data on that exact scenario (using a
> libre= qos middlebox) which I'm still thinking about.
>
> >> >
> >
> > The problem here is that the tim= escale of the scheduling of link bandwidth is
> incompatible with t= he end-to-end TCP congestion control loops. This guarantees
> contr= ol theory instability.
>
> While I don't disagree a cite o= r three would help me. Yes! unstable!
> How unstable?
> > My mental model, originally, was that the terminal scheduling was v= ery
> simple TDMA... which is stable,
> predictable, and wa= steful.
>
>
> >
> >
> > = Now if Starlink had some good, smart people who understand Internet
&g= t; congestion control and queueing theory and control theory in practice (a= nd some of
> those are participating on this list, probably easy to= hire [hint]) ---
>
> I hope they have people on the probl= em.
>
> > the obvious place to incorporate Cake or FQ-c= odel in the path from terminal
> to public Internet would be *in th= e ground station* (or maybe in the satellite, if
> the satellite ac= tually controls the scheduling of access).
>
> This does r= equire an oracle to some extent. I think that the scheduler
> does = need to be on the ground stations, but fq+aqm belongs everywhere.
>=
> >
> >
> > But before we get thrilled a= bout reverse-engineering the Starlink terminal,
> remember, Starlin= k's technology is quite different from a cable data network or a
> = GPON network in its packet architecture. Simple replacement of its software= or
> simple patches probably won't work.
>
> It's= just two easily dissassembled binaries, one for tx, the other for rx.
>
> I have (delusionally) fond memories of porting the decompi= led "empire"
> game around back in the 80s. Not sure how much bette= r the tools have
> got (and this activity is expressly prohibited b= y the starlink eula)
>
> >
> >
> >= ; That said, the talk about breaking into the terminal is quite fascinating= . I
> do believe Starlink has real defense-in-depth, though, so it'= s unlikely that
> Starlink other users are at all Pwned if you do t= hat. Yeah, if its your terminal,
> you can probably sniff your traf= fic and mess up your use. It might be that you can
> disrupt (DoS) = a satellite by "jamming" its "bent-pipe channels". But that's
> nor= mal, nothing out of the ordinary. They will come and get you, and it is lik= ely
> gonna get you jailed for a long time.
>
> Th= eir list of prohibitions and rules for bug bounty security
> resear= chers is reasonable.
>
> The gateway for getting in there = to actually improve the system is
> seemingly insurmountable.
= >
> >
> >
> >
> >
>= > >
> > > (great blackhat talk, btw)
>
&= gt; The one on john deere was also great
>
> https://docto= row.medium.com/this-weekend-i-watched-a-hacker-jailbreak-a-john-deere-tract= or-live-on-stage-febbb0dc5a76
>
> >
> >
> > _______________________________________________
> > S= tarlink mailing list
> > Starlink@lists.bufferbloat.net
>= ; > https://lists.bufferbloat.net/listinfo/starlink
>
>=
>
> --
> FQ World Domination pending: https://bl= og.cerowrt.org/post/state_of_fq_codel/
> Dave T=C3=A4ht CEO, TekLib= re, LLC
>
>
> ------------------------------
>
> Subject: Digest Footer
>
> ______________= _________________________________
> Starlink mailing list
>= Starlink@lists.bufferbloat.net
> https://lists.bufferbloat.net/lis= tinfo/starlink
>
>
> ----------------------------= --
>
> End of Starlink Digest, Vol 17, Issue 16
> = ****************************************
>

=0A
 ------=_20220824142507000000_29345--