From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp101.iad3a.emailsrvr.com (smtp101.iad3a.emailsrvr.com [173.203.187.101]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 15AF93CB64; Fri, 9 Jul 2021 15:31:24 -0400 (EDT) Received: from app36.wa-webapps.iad3a (relay-webapps.rsapps.net [172.27.255.140]) by smtp13.relay.iad3a.emailsrvr.com (SMTP Server) with ESMTP id 2E04A50E9; Fri, 9 Jul 2021 15:31:23 -0400 (EDT) Received: from deepplum.com (localhost.localdomain [127.0.0.1]) by app36.wa-webapps.iad3a (Postfix) with ESMTP id 18A7F603D7; Fri, 9 Jul 2021 15:31:23 -0400 (EDT) Received: by apps.rackspace.com (Authenticated sender: dpreed@deepplum.com, from: dpreed@deepplum.com) with HTTP; Fri, 9 Jul 2021 15:31:23 -0400 (EDT) X-Auth-ID: dpreed@deepplum.com Date: Fri, 9 Jul 2021 15:31:23 -0400 (EDT) From: "David P. Reed" To: "Luca Muscariello" Cc: "Leonard Kleinrock" , starlink@lists.bufferbloat.net, "Make-Wifi-fast" , "Bob McMahon" , "Cake List" , codel@lists.bufferbloat.net, "cerowrt-devel" , "bloat" , "Ben Greear" MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_20210709153123000000_13915" Importance: Normal X-Priority: 3 (Normal) X-Type: html In-Reply-To: References: <1625188609.32718319@apps.rackspace.com> <989de0c1-e06c-cda9-ebe6-1f33df8a4c24@candelatech.com> <1625773080.94974089@apps.rackspace.com> X-Client-IP: 209.6.168.128 Message-ID: <1625859083.09751240@apps.rackspace.com> X-Mailer: webmail/19.0.7-RC X-Classification-ID: 9a7a9cb8-a0df-4471-b57f-d4e1b4c45cab-1-1 Subject: Re: [Cerowrt-devel] Little's Law mea culpa, but not invalidating my main point X-BeenThere: cerowrt-devel@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Development issues regarding the cerowrt test router project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Jul 2021 19:31:24 -0000 ------=_20210709153123000000_13915 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable =0ALen - I admit I made a mistake in challenging Little's Law as being base= d on Poisson processes. It is more general. But it tells you an "average" i= n its base form, and latency averages are not useful for end user applicati= ons.=0A =0AHowever, Little's Law does assume something that is not actually= valid about the kind of distributions seen in the network, and in fact, it= is NOT true that networks converge on Poisson arrival times.=0A =0AThe key= issue is well-described in the sandard analysis of the M/M/1 queue (e.g. [= https://en.wikipedia.org/wiki/M/M/1_queue ]( https://en.wikipedia.org/wiki= /M/M/1_queue )) , which is done only for Poisson processes, and is also lim= ited to "stable" systems. But networks are never stable when fully loaded. = They get unstable and those instabilities persist for a long time in the ne= twork. Instability is at core the underlying *requirement* of the Internet'= s usage.=0A =0ASo specifically: real networks, even large ones, and certain= ly the Internet today, are not asymptotic limits of sums of stationary stoc= hastic arrival processes. Each esternal terminal of any real network has a = real user there, running a real application, and the network is a complex g= raph. This makes it completely unlike a single queue. Even the links within= a network carry a relatively small number of application flows. There's no= ability to apply the Law of Large Numbers to the distributions, because an= y particular path contains only a small number of serialized flows with hig= htly variable rates.=0A =0AHere's an example of what really happens in a re= al network (I've observed this in 5 different cities on ATT's cellular netw= ork, back when it was running Alcatel Lucent HSPA+ gear in those cities).= =0ABut you can see this on any network where transient overload occurs, cre= ating instability.=0A =0A =0AAt 7 AM, the data transmission of the network = is roughty stable. That's because no links are overloaded within the networ= k. Little's Law can tell you by observing the delay and throughput on any p= ath that the average delay in the network is X.=0A =0AContinue sampling del= ay in the network as the day wears on. At about 10 AM, ping delay starts to= soar into the multiple second range. No packers are lost. The peak ping ti= me is about 4000 milliseconds - 4 seconds in most of the networks. This is = in downtown, no radio errors are reported, no link errors.=0ASo it is all q= ueueing delay. =0A =0ANow what Little's law doesn't tell you much about ave= rage delay, because clearly *some* subpiece of the network is fully saturat= ed. But what is interesting here is what is happening and where. You can't = tell what is saturated, and in fact the entire network is quite unstable, b= ecause the peak is constantly varying and you don't know where the throughp= ut is. All the packets are now arriving 4 seconds or so later.=0A =0AWhy is= the situaton not worse than 4 seconds? Well, there are multiple things goi= ng on:=0A =0A1) TCP may be doing a lot of retransmissions (non-Poisson at a= ll, not random either. The arrival process is entirely deterministic in eac= h source, based on the retransmission timeout) or it may not be.=0A =0A2) U= sers are pissed off, because they clicked on a web page, and got nothing ba= ck. They retry on their screen, or they try another site. Meanwhile, the un= derlying TCP connection remains there, pumping the network full of more pac= kets on that old path, which is still backed up with packets that haven't b= een delivered that are sitting in queues. The real arrival process is not P= oisson at all, its a deterministic, repeated retrsnsmission plus a new atte= mpt to connect to a new site.=0A =0A3) When the users get a web page back e= ventually, it is filled with names of other pieces needed to display that w= eb page, which causes some number (often as many as 100) new pages to be fe= tched, ALL at the same time. Certainly not a stochastic process that will j= ust obey the law of large numbers.=0A =0AAll of these things are the result= of initial instability, causing queues to build up.=0A =0ASo what is the s= tate of the system? is it stable? is it stochastic? Is it the sum of enough= stochastic stable flows to average out to Poisson?=0A =0AThe answer is cle= arly NO. Control theory (not queuing theory) suggests that this system is c= ompletely uncontrolled and unstable.=0A =0ASo if the system is in this stat= e, what does Little's Lemma tell us? What is the meaning of that hightly va= riable 4 second delay on ping packets, in terms of average utilizaton of th= e network?=0A =0AWe don't even know what all the users really might need, i= f the system hadn't become unstable, because some users have given up, and = others are trying even harder, and new users are arriving.=0A =0AWhat we do= know, because ATT (at my suggestion) reconfigured their system after blami= ng Apple Computer company for "bugs" in the original iPhone in public, is t= hat simply *dropping* packets sitting in queues more than a couple millisec= onds MADE THE USERS HAPPY. Apparently the required capacity was there all a= long! =0A =0ASo I conclude that the 4 second delay was the largest delay us= ers could barely tolerate before deciding the network was DOWN and going aw= ay. And that the backup was the accumulation of useless packets sitting in = queues because none of the end systems were receiving congestion signals (w= hich for the Internet stack begins with packet dropping).=0A =0AI should sa= y that most operators, and especially ATT in this case, do not measure end-= to-end latency. Instead they use Little's Lemma to query routers for their = current throughput in bits per second, and calculate latency as if Little's= Lemma applied. This results in reports to management that literally say:= =0A =0A The network is not dropping packets, utilization is near 100% on m= any of our switches and routers.=0A =0AAnd management responds, Hooray! Bec= ause utilization of 100% of their hardware is their investors' metric of ma= ximizing profits. The hardware they are operating is fully utilized. No was= te! And users are happy because no packets have been dropped!=0A =0AHmm... = what's wrong with this picture? I can see why Donovan, CTO, would accuse Ap= ple of lousy software that was ruining iPhone user experience! His network= was operating without ANY problems.=0ASo it must be Apple!=0A =0AWell, no.= The entire problem, as we saw when ATT just changed to shorten egress queu= es and drop packets when the egress queues overflowed, was that ATT's netwo= rk was amplifying instability, not at the link level, but at the network le= vel.=0A =0AAnd queueing theory can help with that, but *intro queueing theo= ry* cannot.=0A =0AAnd a big part of that problem is the pervasive belief th= at, at the network boundary, *Poisson arrival* is a reasonable model for us= e in all cases.=0A =0A =0A =0A =0A =0A =0A =0A =0A =0A =0AOn Friday, July 9= , 2021 6:05am, "Luca Muscariello" said:=0A=0A=0A=0A= =0A=0A=0A=0AFor those who might be interested in Little's law=0Athere is a = nice paper by John Little on the occasion =0Aof the 50th anniversary of th= e result.=0A[ https://www.informs.org/Blogs/Operations-Research-Forum/Littl= e-s-Law-as-Viewed-on-its-50th-Anniversary ]( https://www.informs.org/Blogs/= Operations-Research-Forum/Little-s-Law-as-Viewed-on-its-50th-Anniversary )= =0A[ https://www.informs.org/content/download/255808/2414681/file/little_pa= per.pdf ]( https://www.informs.org/content/download/255808/2414681/file/lit= tle_paper.pdf )=0A =0ANice read. =0ALuca =0A =0AP.S. =0AWho has not a copy = of L. Kleinrock's books? I do have and am not ready to lend them!=0A=0AOn F= ri, Jul 9, 2021 at 11:01 AM Leonard Kleinrock <[ lk@cs.ucla.edu ]( mailto:l= k@cs.ucla.edu )> wrote:=0ADavid,=0AI totally appreciate your attention to = when and when not analytical modeling works. Let me clarify a few things fr= om your note.=0AFirst, Little's law (also known as Little=E2=80=99s lemma o= r, as I use in my book, Little=E2=80=99s result) does not assume Poisson ar= rivals - it is good for any arrival process and any service process and is= an equality between time averages. It states that the time average of the= number in a system (for a sample path w) is equal to the average arrival r= ate to the system multiplied by the time-averaged time in the system for th= at sample path. This is often written as NTimeAvg =3D=CE=BB=C2=B7TTimeAv= g . Moreover, if the system is also ergodic, then the time average equals = the ensemble average and we often write it as N =CC=84 =3D =CE=BB T =CC=84 = . In any case, this requires neither Poisson arrivals nor exponential serv= ice times. =0A =0AQueueing theorists often do study the case of Poisson ar= rivals. True, it makes the analysis easier, yet there is a better reason i= t is often used, and that is because the sum of a large number of independe= nt stationary renewal processes approaches a Poisson process. So nature of= ten gives us Poisson arrivals. =0ABest,=0ALen=0A=0A=0AOn Jul 8, 2021, at 1= 2:38 PM, David P. Reed <[ dpreed@deepplum.com ]( mailto:dpreed@deepplum.com= )> wrote:=0A=0A=0AI will tell you flat out that the arrival time distribut= ion assumption made by Little's Lemma that allows "estimation of queue dept= h" is totally unreasonable on ANY Internet in practice.=0A =0AThe assumptio= n is a Poisson Arrival Process. In reality, traffic arrivals in real intern= et applications are extremely far from Poisson, and, of course, using TCP w= indowing, become highly intercorrelated with crossing traffic that shares t= he same queue.=0A =0ASo, as I've tried to tell many, many net-heads (people= who ignore applications layer behavior, like the people that think latency= doesn't matter to end users, only throughput), end-to-end packet arrival t= imes on a practical network are incredibly far from Poisson - and they are = more like fractal probability distributions, very irregular at all scales o= f time.=0A =0ASo, the idea that iperf can estimate queue depth by Little's = Lemma by just measuring saturation of capacity of a path is bogus.The less = Poisson, the worse the estimate gets, by a huge factor.=0A =0A =0AWhere doe= s the Poisson assumption come from? Well, like many theorems, it is the si= mplest tractable closed form solution - it creates a simplified view, by be= ing a "single-parameter" distribution (the parameter is called lambda for a= Poisson distribution). And the analysis of a simple queue with poisson ar= rival distribution and a static, fixed service time is the first interestin= g Queueing Theory example in most textbooks. It is suggestive of an interes= ting phenomenon, but it does NOT characterize any real system.=0A =0AIt's t= he queueing theory equivalent of "First, we assume a spherical cow...". in = doing an example in a freshman physics class.=0A =0AUnfortunately, most net= working engineers understand neither queuing theory nor application network= ing usage in interactive applications. Which makes them arrogant. They assu= me all distributions are poisson!=0A =0A =0AOn Tuesday, July 6, 2021 9:46am= , "Ben Greear" <[ greearb@candelatech.com ]( mailto:greearb@candelatech.com= )> said:=0A=0A=0A=0A> Hello,=0A> =0A> I am interested to hear wish lists f= or network testing features. We make test=0A> equipment, supporting lots=0A= > of wifi stations and a distributed architecture, with built-in udp, tcp, = ipv6,=0A> http, ... protocols,=0A> and open to creating/improving some of o= ur automated tests.=0A> =0A> I know Dave has some test scripts already, so = I'm not necessarily looking to=0A> reimplement that,=0A> but more fishing f= or other/new ideas.=0A> =0A> Thanks,=0A> Ben=0A> =0A> On 7/2/21 4:28 PM, Bo= b McMahon wrote:=0A> > I think we need the language of math here. It seems = like the network=0A> power metric, introduced by Kleinrock and Jaffe in the= late 70s, is something=0A> useful.=0A> > Effective end/end queue depths pe= r Little's law also seems useful. Both are=0A> available in iperf 2 from a = test perspective. Repurposing test techniques to=0A> actual=0A> > traffic c= ould be useful. Hence the question around what exact telemetry=0A> is usefu= l to apps making socket write() and read() calls.=0A> >=0A> > Bob=0A> >=0A>= > On Fri, Jul 2, 2021 at 10:07 AM Dave Taht <[ dave.taht@gmail.com ]( mail= to:dave.taht@gmail.com )=0A> <[ mailto:dave.taht@gmail.com ]( mailto:dave.t= aht@gmail.com )>> wrote:=0A> >=0A> > In terms of trying to find "Quality" I= have tried to encourage folk to=0A> > both read "zen and the art of motorc= ycle maintenance"[0], and Deming's=0A> > work on "total quality management"= .=0A> >=0A> > My own slice at this network, computer and lifestyle "issue" = is aiming=0A> > for "imperceptible latency" in all things. [1]. There's a l= ot of=0A> > fallout from that in terms of not just addressing queuing delay= , but=0A> > caching, prefetching, and learning more about what a user reall= y needs=0A> > (as opposed to wants) to know via intelligent agents.=0A> >= =0A> > [0] If you want to get depressed, read Pirsig's successor to "zen...= ",=0A> > lila, which is in part about what happens when an engineer hits an= =0A> > insoluble problem.=0A> > [1] [ https://www.internetsociety.org/event= s/latency2013/ ]( https://www.internetsociety.org/events/latency2013/ )=0A>= <[ https://www.internetsociety.org/events/latency2013/ ]( https://www.inte= rnetsociety.org/events/latency2013/ )>=0A> >=0A> >=0A> >=0A> > On Thu, Jul = 1, 2021 at 6:16 PM David P. Reed <[ dpreed@deepplum.com ]( mailto:dpreed@de= epplum.com )=0A> <[ mailto:dpreed@deepplum.com ]( mailto:dpreed@deepplum.co= m )>> wrote:=0A> > >=0A> > > Well, nice that the folks doing the conference= are willing to=0A> consider that quality of user experience has little to= do with signalling rate at=0A> the=0A> > physical layer or throughput of F= TP transfers.=0A> > >=0A> > >=0A> > >=0A> > > But honestly, the fact that t= hey call the problem "network quality"=0A> suggests that they REALLY, REALL= Y don't understand the Internet isn't the hardware=0A> or=0A> > the routers= or even the routing algorithms *to its users*.=0A> > >=0A> > >=0A> > >=0A>= > > By ignoring the diversity of applications now and in the future,=0A> a= nd the fact that we DON'T KNOW what will be coming up, this conference will= =0A> likely fall=0A> > into the usual trap that net-heads fall into - optim= izing for some=0A> imaginary reality that doesn't exist, and in fact will p= robably never be what=0A> users=0A> > actually will do given the chance.=0A= > > >=0A> > >=0A> > >=0A> > > I saw this issue in 1976 in the group develop= ing the original=0A> Internet protocols - a desire to put *into the network= * special tricks to optimize=0A> ASR33=0A> > logins to remote computers fro= m terminal concentrators (aka remote=0A> login), bulk file transfers betwee= n file systems on different time-sharing=0A> systems, and=0A> > "sessions" = (virtual circuits) that required logins. And then trying to=0A> exploit und= erlying "multicast" by building it into the IP layer, because someone=0A> >= thought that TV broadcast would be the dominant application.=0A> > >=0A> >= >=0A> > >=0A> > > Frankly, to think of "quality" as something that can be = "provided"=0A> by "the network" misses the entire point of "end-to-end argu= ment in system=0A> design".=0A> > Quality is not a property defined or crea= ted by The Network. If you want=0A> to talk about Quality, you need to talk= about users - all the users at all times,=0A> > now and into the future, a= nd that's something you can't do if you don't=0A> bother to include current= and future users talking about what they might expect=0A> to=0A> > experie= nce that they don't experience.=0A> > >=0A> > >=0A> > >=0A> > > There was m= uch fighting back in 1976 that basically involved=0A> "network experts" say= ing that the network was the place to "solve" such issues as=0A> quality,= =0A> > so applications could avoid having to solve such issues.=0A> > >=0A>= > >=0A> > >=0A> > > What some of us managed to do was to argue that you ca= n't "solve"=0A> such issues. All you can do is provide a framework that ena= bles different uses to=0A> > *cooperate* in some way.=0A> > >=0A> > >=0A> >= >=0A> > > Which is why the Internet drops packets rather than queueing the= m,=0A> and why diffserv cannot work.=0A> > >=0A> > > (I know the latter is = conftroversial, but at the moment, ALL of=0A> diffserv attempts to talk abo= ut end-to-end applicaiton specific metrics, but=0A> never, ever=0A> > expla= ins what the diffserv control points actually do w.r.t. what the IP=0A> lay= er can actually control. So it is meaningless - another violation of the=0A= > > so-called end-to-end principle).=0A> > >=0A> > >=0A> > >=0A> > > Networ= ks are about getting packets from here to there, multiplexing=0A> the under= lying resources. That's it. Quality is a whole different thing. Quality=0A>= can=0A> > be improved by end-to-end approaches, if the underlying network = provides=0A> some kind of thing that actually creates a way for end-to-end = applications to=0A> > affect queueing and routing decisions, and more impor= tantly getting=0A> "telemetry" from the network regarding what is actually = going on with the other=0A> > end-to-end users sharing the infrastructure.= =0A> > >=0A> > >=0A> > >=0A> > > This conference won't talk about it this w= ay. So don't waste your=0A> time.=0A> > >=0A> > >=0A> > >=0A> > >=0A> > >= =0A> > >=0A> > >=0A> > > On Wednesday, June 30, 2021 8:12pm, "Dave Taht"=0A= > <[ dave.taht@gmail.com ]( mailto:dave.taht@gmail.com ) <[ mailto:dave.tah= t@gmail.com ]( mailto:dave.taht@gmail.com )>> said:=0A> > >=0A> > > > The p= rogram committee members are *amazing*. Perhaps, finally,=0A> we can=0A> > = > > move the bar for the internet's quality metrics past endless,=0A> blind= =0A> > > > repetitions of speedtest.=0A> > > >=0A> > > > For complete detai= ls, please see:=0A> > > > [ https://www.iab.org/activities/workshops/networ= k-quality/ ]( https://www.iab.org/activities/workshops/network-quality/ )= =0A> <[ https://www.iab.org/activities/workshops/network-quality/ ]( https:= //www.iab.org/activities/workshops/network-quality/ )>=0A> > > >=0A> > > > = Submissions Due: Monday 2nd August 2021, midnight AOE=0A> (Anywhere On Eart= h)=0A> > > > Invitations Issued by: Monday 16th August 2021=0A> > > >=0A> >= > > Workshop Date: This will be a virtual workshop, spread over=0A> three = days:=0A> > > >=0A> > > > 1400-1800 UTC Tue 14th September 2021=0A> > > > 1= 400-1800 UTC Wed 15th September 2021=0A> > > > 1400-1800 UTC Thu 16th Septe= mber 2021=0A> > > >=0A> > > > Workshop co-chairs: Wes Hardaker, Evgeny Khor= ov, Omer Shapira=0A> > > >=0A> > > > The Program Committee members:=0A> > >= >=0A> > > > Jari Arkko, Olivier Bonaventure, Vint Cerf, Stuart Cheshire,= =0A> Sam=0A> > > > Crowford, Nick Feamster, Jim Gettys, Toke Hoiland-Jorgen= sen,=0A> Geoff=0A> > > > Huston, Cullen Jennings, Katarzyna Kosek-Szott, Mi= rja=0A> Kuehlewind,=0A> > > > Jason Livingood, Matt Mathias, Randall Meyer,= Kathleen=0A> Nichols,=0A> > > > Christoph Paasch, Tommy Pauly, Greg White,= Keith Winstein.=0A> > > >=0A> > > > Send Submissions to: [ network-quality= -workshop-pc@iab.org ]( mailto:network-quality-workshop-pc@iab.org )=0A> <[= mailto:network-quality-workshop-pc@iab.org ]( mailto:network-quality-works= hop-pc@iab.org )>.=0A> > > >=0A> > > > Position papers from academia, indus= try, the open source=0A> community and=0A> > > > others that focus on measu= rements, experiences, observations=0A> and=0A> > > > advice for the future = are welcome. Papers that reflect=0A> experience=0A> > > > based on deployed= services are especially welcome. The=0A> organizers=0A> > > > understand t= hat specific actions taken by operators are=0A> unlikely to be=0A> > > > di= scussed in detail, so papers discussing general categories=0A> of=0A> > > >= actions and issues without naming specific technologies,=0A> products, or= =0A> > > > other players in the ecosystem are expected. Papers should not= =0A> focus=0A> > > > on specific protocol solutions.=0A> > > >=0A> > > > Th= e workshop will be by invitation only. Those wishing to=0A> attend=0A> > > = > should submit a position paper to the address above; it may=0A> take the= =0A> > > > form of an Internet-Draft.=0A> > > >=0A> > > > All inputs submit= ted and considered relevant will be published=0A> on the=0A> > > > workshop= website. The organisers will decide whom to invite=0A> based on=0A> > > > = the submissions received. Sessions will be organized according=0A> to=0A> >= > > content, and not every accepted submission or invited attendee=0A> wil= l=0A> > > > have an opportunity to present as the intent is to foster=0A> d= iscussion=0A> > > > and not simply to have a sequence of presentations.=0A>= > > >=0A> > > > Position papers from those not planning to attend the virt= ual=0A> sessions=0A> > > > themselves are also encouraged. A workshop repor= t will be=0A> published=0A> > > > afterwards.=0A> > > >=0A> > > > Overview:= =0A> > > >=0A> > > > "We believe that one of the major factors behind this = lack of=0A> progress=0A> > > > is the popular perception that throughput is= the often sole=0A> measure of=0A> > > > the quality of Internet connectivi= ty. With such narrow focus,=0A> people=0A> > > > don=E2=80=99t consider que= stions such as:=0A> > > >=0A> > > > What is the latency under typical worki= ng conditions?=0A> > > > How reliable is the connectivity across longer tim= e periods?=0A> > > > Does the network allow the use of a broad range of pro= tocols?=0A> > > > What services can be run by clients of the network?=0A> >= > > What kind of IPv4, NAT or IPv6 connectivity is offered, and=0A> are th= ere firewalls?=0A> > > > What security mechanisms are available for local s= ervices,=0A> such as DNS?=0A> > > > To what degree are the privacy, confide= ntiality, integrity=0A> and=0A> > > > authenticity of user communications g= uarded?=0A> > > >=0A> > > > Improving these aspects of network quality will= likely depend=0A> on=0A> > > > measurement and exposing metrics to all inv= olved parties,=0A> including to=0A> > > > end users in a meaningful way. Su= ch measurements and exposure=0A> of the=0A> > > > right metrics will allow = service providers and network=0A> operators to=0A> > > > focus on the aspec= ts that impacts the users=E2=80=99 experience=0A> most and at=0A> > > > the= same time empowers users to choose the Internet service=0A> that will=0A> = > > > give them the best experience."=0A> > > >=0A> > > >=0A> > > > --=0A> = > > > Latest Podcast:=0A> > > >=0A> [ https://www.linkedin.com/feed/update/= urn:li:activity:6791014284936785920/ ]( https://www.linkedin.com/feed/updat= e/urn:li:activity:6791014284936785920/ )=0A> <[ https://www.linkedin.com/fe= ed/update/urn:li:activity:6791014284936785920/ ]( https://www.linkedin.com/= feed/update/urn:li:activity:6791014284936785920/ )>=0A> > > >=0A> > > > Dav= e T=C3=A4ht CTO, TekLibre, LLC=0A> > > > __________________________________= _____________=0A> > > > Cerowrt-devel mailing list=0A> > > > [ Cerowrt-deve= l@lists.bufferbloat.net ]( mailto:Cerowrt-devel@lists.bufferbloat.net )=0A>= <[ mailto:Cerowrt-devel@lists.bufferbloat.net ]( mailto:Cerowrt-devel@list= s.bufferbloat.net )>=0A> > > > [ https://lists.bufferbloat.net/listinfo/cer= owrt-devel ]( https://lists.bufferbloat.net/listinfo/cerowrt-devel )=0A> <[= https://lists.bufferbloat.net/listinfo/cerowrt-devel ]( https://lists.buff= erbloat.net/listinfo/cerowrt-devel )>=0A> > > >=0A> >=0A> >=0A> >=0A> > --= =0A> > Latest Podcast:=0A> > [ https://www.linkedin.com/feed/update/urn:li:= activity:6791014284936785920/ ]( https://www.linkedin.com/feed/update/urn:l= i:activity:6791014284936785920/ )=0A> <[ https://www.linkedin.com/feed/upda= te/urn:li:activity:6791014284936785920/ ]( https://www.linkedin.com/feed/up= date/urn:li:activity:6791014284936785920/ )>=0A> >=0A> > Dave T=C3=A4ht CTO= , TekLibre, LLC=0A> > _______________________________________________=0A> >= Make-wifi-fast mailing list=0A> > [ Make-wifi-fast@lists.bufferbloat.net ]= ( mailto:Make-wifi-fast@lists.bufferbloat.net )=0A> <[ mailto:Make-wifi-fas= t@lists.bufferbloat.net ]( mailto:Make-wifi-fast@lists.bufferbloat.net )>= =0A> > [ https://lists.bufferbloat.net/listinfo/make-wifi-fast ]( https://l= ists.bufferbloat.net/listinfo/make-wifi-fast )=0A> <[ https://lists.bufferb= loat.net/listinfo/make-wifi-fast ]( https://lists.bufferbloat.net/listinfo/= make-wifi-fast )>=0A> >=0A> >=0A> > This electronic communication and the i= nformation and any files transmitted=0A> with it, or attached to it, are co= nfidential and are intended solely for the use=0A> of=0A> > the individual = or entity to whom it is addressed and may contain information=0A> that is c= onfidential, legally privileged, protected by privacy laws, or otherwise=0A= > > restricted from disclosure to anyone else. If you are not the intended= =0A> recipient or the person responsible for delivering the e-mail to the i= ntended=0A> recipient,=0A> > you are hereby notified that any use, copying,= distributing, dissemination,=0A> forwarding, printing, or copying of this = e-mail is strictly prohibited. If you=0A> > received this e-mail in error, = please return the e-mail to the sender, delete=0A> it from your computer, a= nd destroy any printed copy of it.=0A> >=0A> > ____________________________= ___________________=0A> > Starlink mailing list=0A> > [ Starlink@lists.buff= erbloat.net ]( mailto:Starlink@lists.bufferbloat.net )=0A> > [ https://list= s.bufferbloat.net/listinfo/starlink ]( https://lists.bufferbloat.net/listin= fo/starlink )=0A> >=0A> =0A> =0A> --=0A> Ben Greear <[ greearb@candelatech.= com ]( mailto:greearb@candelatech.com )>=0A> Candela Technologies Inc [ htt= p://www.candelatech.com ]( http://www.candelatech.com )=0A>________________= _______________________________=0AStarlink mailing list=0A[ Starlink@lists.= bufferbloat.net ]( mailto:Starlink@lists.bufferbloat.net )=0A[ https://list= s.bufferbloat.net/listinfo/starlink ]( https://lists.bufferbloat.net/listin= fo/starlink )_______________________________________________=0A Make-wifi-f= ast mailing list=0A[ Make-wifi-fast@lists.bufferbloat.net ]( mailto:Make-wi= fi-fast@lists.bufferbloat.net )=0A[ https://lists.bufferbloat.net/listinfo/= make-wifi-fast ]( https://lists.bufferbloat.net/listinfo/make-wifi-fast ) ------=_20210709153123000000_13915 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

Len - I admit I made a= mistake in challenging Little's Law as being based on Poisson processes. I= t is more general. But it tells you an "average" in its base form, and late= ncy averages are not useful for end user applications.

=0A

 

=0A

However, Little's Law does assume = something that is not actually valid about the kind of distributions seen i= n the network, and in fact, it is NOT true that networks converge on Poisso= n arrival times.

=0A

 

=0A

The key issue is well-described in the sandard analysis of the M/M/1 que= ue (e.g. https://en.w= ikipedia.org/wiki/M/M/1_queue) , which is done only for Poisson process= es, and is also limited to "stable" systems. But networks are never stable = when fully loaded. They get unstable and those instabilities persist for a = long time in the network. Instability is at core the underlying *requiremen= t* of the Internet's usage.

=0A

 

=0A

So specifically: real networks, even large ones, and certainl= y the Internet today, are not asymptotic limits of sums of stationary stoch= astic arrival processes. Each esternal terminal of any real network has a r= eal user there, running a real application, and the network is a complex gr= aph. This makes it completely unlike a single queue. Even the links within = a network carry a relatively small number of application flows. There's no = ability to apply the Law of Large Numbers to the distributions, because any= particular path contains only a small number of serialized flows with high= tly variable rates.

=0A

 

=0A

Here's an example of what really happens in a real network (I've obse= rved this in 5 different cities on ATT's cellular network, back when it was= running Alcatel Lucent HSPA+ gear in those cities).

=0A

But you can see this on any network where transient overload occurs, c= reating instability.

=0A

 

=0A

 

=0A

At 7 AM, the data transmission of = the network is roughty stable. That's because no links are overloaded withi= n the network. Little's Law can tell you by observing the delay and through= put on any path that the average delay in the network is X.

=0A

 

=0A

Continue sampling delay in t= he network as the day wears on. At about 10 AM, ping delay starts to soar i= nto the multiple second range. No packers are lost. The peak ping time is a= bout 4000 milliseconds - 4 seconds in most of the networks. This is in down= town, no radio errors are reported, no link errors.

=0A

So it is all queueing delay. 

=0A

 =0A

Now what Little's law doesn't tell you much about = average delay, because clearly *some* subpiece of the network is fully satu= rated. But what is interesting here is what is happening and where. You can= 't tell what is saturated, and in fact the entire network is quite unstable= , because the peak is constantly varying and you don't know where the throu= ghput is. All the packets are now arriving 4 seconds or so later.

=0A

 

=0A

Why is the situaton not= worse than 4 seconds? Well, there are multiple things going on:

=0A

 

=0A

1) TCP may be doing a lo= t of retransmissions (non-Poisson at all, not random either. The arrival pr= ocess is entirely deterministic in each source, based on the retransmission= timeout) or it may not be.

=0A

 

=0A

2) Users are pissed off, because they clicked on a web page, = and got nothing back. They retry on their screen, or they try another site.= Meanwhile, the underlying TCP connection remains there, pumping the networ= k full of more packets on that old path, which is still backed up with pack= ets that haven't been delivered that are sitting in queues. The real arriva= l process is not Poisson at all, its a deterministic, repeated retrsnsmissi= on plus a new attempt to connect to a new site.

=0A

=  

=0A

3) When the users get a web page back eve= ntually, it is filled with names of other pieces needed to display that web= page, which causes some number (often as many as 100) new pages to be fetc= hed, ALL at the same time. Certainly not a stochastic process that will jus= t obey the law of large numbers.

=0A

 

=0AAll of these things are the result of initial instabilit= y, causing queues to build up.

=0A

 

=0A

So what is the state of the system? is it stable? is it st= ochastic? Is it the sum of enough stochastic stable flows to average out to= Poisson?

=0A

 

=0A

The = answer is clearly NO. Control theory (not queuing theory) suggests that thi= s system is completely uncontrolled and unstable.

=0A

So if the system is in this state, what= does Little's Lemma tell us? What is the meaning of that hightly variable = 4 second delay on ping packets, in terms of average utilizaton of the netwo= rk?

=0A

 

=0A

We don't e= ven know what all the users really might need, if the system hadn't become = unstable, because some users have given up, and others are trying even hard= er, and new users are arriving.

=0A

 

=0A

What we do know, because ATT (at my suggestion) reconfigu= red their system after blaming Apple Computer company for "bugs" in the ori= ginal iPhone in public, is that simply *dropping* packets sitting in queues= more than a couple milliseconds MADE THE USERS HAPPY. Apparently the requi= red capacity was there all along! 

=0A

 =0A

So I conclude that the 4 second delay was the lar= gest delay users could barely tolerate before deciding the network was DOWN= and going away. And that the backup was the accumulation of useless packet= s sitting in queues because none of the end systems were receiving congesti= on signals (which for the Internet stack begins with packet dropping).

= =0A

 

=0A

I should say that= most operators, and especially ATT in this case, do not measure end-to-end= latency. Instead they use Little's Lemma to query routers for their curren= t throughput in bits per second, and calculate latency as if Little's Lemma= applied. This results in reports to management that literally say:

=0A<= p style=3D"margin:0;padding:0;font-family: arial; font-size: 10pt; overflow= -wrap: break-word;"> 

=0A

  The network is= not dropping packets, utilization is near 100% on many of our switches and= routers.

=0A

 

=0A

And = management responds, Hooray! Because utilization of 100% of their hardware = is their investors' metric of maximizing profits. The hardware they are ope= rating is fully utilized. No waste! And users are happy because no packets = have been dropped!

=0A

 

=0A

Hmm... what's wrong with this picture? I can see why Donovan, CTO, wou= ld accuse Apple of lousy software that was ruining iPhone user experience!&= nbsp; His network was operating without ANY problems.

=0A

So it must be Apple!

=0A

 

=0A

Well, no. The entire problem, as we saw when ATT just changed = to shorten egress queues and drop packets when the egress queues overflowed= , was that ATT's network was amplifying instability, not at the link level,= but at the network level.

=0A

 

=0A

And queueing theory can help with that, but *intro queueing th= eory* cannot.

=0A

 

=0A

= And a big part of that problem is the pervasive belief that, at the network= boundary, *Poisson arrival* is a reasonable model for use in all cases.=0A

 

=0A

 

=0A

 

=0A

 

=0A

 

=0A

 

=0A

 

=0A

 

=0A

&= nbsp;

=0A

 

=0A

On Frida= y, July 9, 2021 6:05am, "Luca Muscariello" <muscariello@ieee.org> sai= d:

=0A
=0A
= =0A
=0A
=0A
=0A
For those who might be= interested in Little's law
=0A
there is a nice paper by John Little on the occasio= n 
=0A
of the 50th anniversary  of the result.
=0A
https://www.informs.org/Blogs/Operations-Research-Forum/Little-s-= Law-as-Viewed-on-its-50th-Anniversary
=0A
https://www.info= rms.org/content/download/255808/2414681/file/little_paper.pdf=0A
=  
=0A
Nice read. 
=0A
Luca 
=0A
 =
=0A
P.S. 
=0A
Who has not a copy of L. Kleinrock's books? I do hav= e and am not ready to lend them!
=0A
=0A
=0A
On Fri, Jul 9, 2021 at 11= :01 AM Leonard Kleinrock <lk@cs.ucla.e= du> wrote:
=0A
= =0A
David,=0A
I totally apprec= iate  your attention to when and when not analytical modeling works. L= et me clarify a few things from your note.
=0A
First, Little's law= (also known as Little=E2=80=99s lemma or, as I use in my book, Little=E2= =80=99s result) does not assume Poisson arrivals -  it is good for any arrival process and any service process and is an equalit= y between time averages.  It states that the time average of the numbe= r in a system (for a sample path w) is equal to the average arrival rate to the system multi= plied by the time-averaged time in the system for that sample path.  T= his is often written as   NTimeAvg =3D=CE=BB=C2=B7= TTimeAvg .  Moreover, if the system is also ergodic, then the time average equ= als the ensemble average and we often write it as N =CC=84 =3D =CE=BB T =CC=84 . =  In any case, this requires neither Poisson arrivals nor exponential s= ervice times.  
=0A
 
=0A
Queueing theorists often do study the case of= Poisson arrivals.  True, it makes the analysis easier, yet there is a= better reason it is often used, and that is because the sum of a large num= ber of independent stationary renewal processes approaches a Poisson proces= s.  So nature often gives us Poisson arrivals.  
=0A
Bes= t,
=0A
Len
=0A
=0A
=0A
=0A
On Jul 8, 2= 021, at 12:38 PM, David P. Reed <dpreed@deepplum.com> wrote:
=0A
=0A
= =0A
I will tell you flat out that the arrival time distribution assumpti= on made by Little's Lemma that allows "estimation of queue depth" is totall= y unreasonable on ANY Internet in practice.
=0A

&n= bsp;

=0A
The assumption is a Poisson Arrival Process. In reality, tra= ffic arrivals in real internet applications are extremely far from Poisson,= and, of course, using TCP windowing, become highly intercorrelated with cr= ossing traffic that shares the same queue.
=0A

&nb= sp;

=0A
So, as I've tried to tell many, many net-heads (people who ig= nore applications layer behavior, like the people that think latency doesn'= t matter to end users, only throughput), end-to-end packet arrival times on= a practical network are incredibly far from Poisson - and they are more li= ke fractal probability distributions, very irregular at all scales of time.=
=0A

 

=0A
So, the idea that iperf can = estimate queue depth by Little's Lemma by just measuring saturation of capa= city of a path is bogus.The less Poisson, the worse the estimate gets, by a= huge factor.
=0A

 

=0A

Where does the Poisson assumption come from?  Well,= like many theorems, it is the simplest tractable closed form solution - it= creates a simplified view, by being a "single-parameter" distribution (the= parameter is called lambda for a Poisson distribution).  And the anal= ysis of a simple queue with poisson arrival distribution and a static, fixe= d service time is the first interesting Queueing Theory example in most tex= tbooks. It is suggestive of an interesting phenomenon, but it does NOT char= acterize any real system.

=0A

 

=0A
It'= s the queueing theory equivalent of "First, we assume a spherical cow...". = in doing an example in a freshman physics class.
=0A

 

=0A
Unfortunately, most networking engineers understand nei= ther queuing theory nor application networking usage in interactive applica= tions. Which makes them arrogant. They assume all distributions are poisson= !
=0A

 

=0A

 

= =0A
On Tuesday, July 6, 2021 9:46am, "Ben Greear" <greearb@candelatech.com> = said:

=0A
=0A
> Hello,
>
> I am interested to hear w= ish lists for network testing features. We make test
> equipment, s= upporting lots
> of wifi stations and a distributed architecture, w= ith built-in udp, tcp, ipv6,
> http, ... protocols,
> and o= pen to creating/improving some of our automated tests.
>
>= I know Dave has some test scripts already, so I'm not necessarily looking = to
> reimplement that,
> but more fishing for other/new ide= as.
>
> Thanks,
> Ben
>
> On 7/2/= 21 4:28 PM, Bob McMahon wrote:
> > I think we need the language&= nbsp;of math here. It seems like the network
> power metric, introd= uced by Kleinrock and Jaffe in the late 70s, is something
> us= eful.
> > Effective end/end queue depths per Little's law also s= eems useful. Both are
> available in iperf 2 from a test perspectiv= e. Repurposing test techniques to
> actual
> > traffic c= ould be useful. Hence the question around what exact telemetry
&g= t; is useful to apps making socket write() and read() calls.
> >=
> > Bob
> >
> > On Fri, Jul 2, 2021 at 10= :07 AM Dave Taht <dave.taht@gmail.com
> <mailto:dave.taht@gmail.com>> wrote:
&= gt; >
> > In terms of trying to find "Quality" I have tried t= o encourage folk to
> > both read "zen and the art of motorcycle= maintenance"[0], and Deming's
> > work on "total quality manage= ment".
> >
> > My own slice at this network, computer= and lifestyle "issue" is aiming
> > for "imperceptible latency"= in all things. [1]. There's a lot of
> > fallout from that in t= erms of not just addressing queuing delay, but
> > caching, pref= etching, and learning more about what a user really needs
> > (a= s opposed to wants) to know via intelligent agents.
> >
>= ; > [0] If you want to get depressed, read Pirsig's successor to "zen...= ",
> > lila, which is in part about what happens when an enginee= r hits an
> > insoluble problem.
> > [1] https= ://www.internetsociety.org/events/latency2013/
> <htt= ps://www.internetsociety.org/events/latency2013/>
> >
> >
> >
> > On Thu, Jul 1, 2021 at 6:16 PM Da= vid P. Reed <dp= reed@deepplum.com
> <mailto:dpreed@deepplum.com>> wrote:
> &= gt; >
> > > Well, nice that the folks doing the conference=   are willing to
> consider that quality of user experience ha= s little to do with signalling rate at
> the
> > physica= l layer or throughput of FTP transfers.
> > >
> > = >
> > >
> > > But honestly, the fact that th= ey call the problem "network quality"
> suggests that they REALLY, = REALLY don't understand the Internet isn't the hardware
> or
&= gt; > the routers or even the routing algorithms *to its users*.
&g= t; > >
> > >
> > >
> > > By= ignoring the diversity of applications now and in the future,
> an= d the fact that we DON'T KNOW what will be coming up, this conference will<= br />> likely fall
> > into the usual trap that net-heads fal= l into - optimizing for some
> imaginary reality that doesn't exist= , and in fact will probably never be what
> users
> > ac= tually will do given the chance.
> > >
> > >> > >
> > > I saw this issue in 1976 in the group= developing the original
> Internet protocols - a desire to put *in= to the network* special tricks to optimize
> ASR33
> > l= ogins to remote computers from terminal concentrators (aka remote
>= login), bulk file transfers between file systems on different time-sharing=
> systems, and
> > "sessions" (virtual circuits) that r= equired logins. And then trying to
> exploit underlying "multicast"= by building it into the IP layer, because someone
> > thought t= hat TV broadcast would be the dominant application.
> > >
> > >
> > >
> > > Frankly, to think = of "quality" as something that can be "provided"
> by "the network"= misses the entire point of "end-to-end argument in system
> design= ".
> > Quality is not a property defined or created by The Netwo= rk. If you want
> to talk about Quality, you need to talk about use= rs - all the users at all times,
> > now and into the future, an= d that's something you can't do if you don't
> bother to include cu= rrent and future users talking about what they might expect
> to> > experience that they don't experience.
> > >
> > >
> > >
> > > There was much fig= hting back in 1976 that basically involved
> "network experts" sayi= ng that the network was the place to "solve" such issues as
> quali= ty,
> > so applications could avoid having to solve such issues.=
> > >
> > >
> > >
> >= > What some of us managed to do was to argue that you can't "solve"
> such issues. All you can do is provide a framework that enables diff= erent uses to
> > *cooperate* in some way.
> > >> > >
> > >
> > > Which is why the= Internet drops packets rather than queueing them,
> and why diffse= rv cannot work.
> > >
> > > (I know the latter = is conftroversial, but at the moment, ALL of
> diffserv attempts to= talk about end-to-end applicaiton specific metrics, but
> never, e= ver
> > explains what the diffserv control points actually do w.= r.t. what the IP
> layer can actually control. So it is meaningless= - another violation of the
> > so-called end-to-end principle).=
> > >
> > >
> > >
> >= > Networks are about getting packets from here to there, multiplexing> the underlying resources. That's it. Quality is a whole different = thing. Quality
> can
> > be improved by end-to-end appro= aches, if the underlying network provides
> some kind of thing that= actually creates a way for end-to-end applications to
> > affec= t queueing and routing decisions, and more importantly getting
> "t= elemetry" from the network regarding what is actually going on with the oth= er
> > end-to-end users sharing the infrastructure.
> &g= t; >
> > >
> > >
> > > This co= nference won't talk about it this way. So don't waste your
> time.<= br />> > >
> > >
> > >
> > = >
> > >
> > >
> > >
> = > > On Wednesday, June 30, 2021 8:12pm, "Dave Taht"
> <dave.taht@gmail.com <mailto:dave= .taht@gmail.com>> said:
> > >
> > > &= gt; The program committee members are *amazing*. Perhaps, finally,
>= ; we can
> > > > move the bar for the internet's quality m= etrics past endless,
> blind
> > > > repetitions o= f speedtest.
> > > >
> > > > For complete= details, please see:
> > > > https://www.iab= .org/activities/workshops/network-quality/
> <h= ttps://www.iab.org/activities/workshops/network-quality/>
> = > > >
> > > > Submissions Due: Monday 2nd August = 2021, midnight AOE
> (Anywhere On Earth)
> > > > I= nvitations Issued by: Monday 16th August 2021
> > > >
> > > > Workshop Date: This will be a virtual workshop, spread= over
> three days:
> > > >
> > > &g= t; 1400-1800 UTC Tue 14th September 2021
> > > > 1400-1800= UTC Wed 15th September 2021
> > > > 1400-1800 UTC Thu 16t= h September 2021
> > > >
> > > > Workshop= co-chairs: Wes Hardaker, Evgeny Khorov, Omer Shapira
> > > &= gt;
> > > > The Program Committee members:
> > = > >
> > > > Jari Arkko, Olivier Bonaventure, Vint Ce= rf, Stuart Cheshire,
> Sam
> > > > Crowford, Nick = Feamster, Jim Gettys, Toke Hoiland-Jorgensen,
> Geoff
> >= ; > > Huston, Cullen Jennings, Katarzyna Kosek-Szott, Mirja
>= Kuehlewind,
> > > > Jason Livingood, Matt Mathias, Randal= l Meyer, Kathleen
> Nichols,
> > > > Christoph Paa= sch, Tommy Pauly, Greg White, Keith Winstein.
> > > >
> > > > Send Submissions to: network-quality-workshop-pc@iab.org=
> <mailto:network-quality-workshop-pc@iab.org>.
&= gt; > > >
> > > > Position papers from academia, = industry, the open source
> community and
> > > > = others that focus on measurements, experiences, observations
> and<= br />> > > > advice for the future are welcome. Papers that ref= lect
> experience
> > > > based on deployed servic= es are especially welcome. The
> organizers
> > > >= ; understand that specific actions taken by operators are
> unlikel= y to be
> > > > discussed in detail, so papers discussing = general categories
> of
> > > > actions and issues= without naming specific technologies,
> products, or
> >= ; > > other players in the ecosystem are expected. Papers should not<= br />> focus
> > > > on specific protocol solutions.> > > >
> > > > The workshop will be by inv= itation only. Those wishing to
> attend
> > > > sh= ould submit a position paper to the address above; it may
> take th= e
> > > > form of an Internet-Draft.
> > > &= gt;
> > > > All inputs submitted and considered relevant w= ill be published
> on the
> > > > workshop website= . The organisers will decide whom to invite
> based on
> &g= t; > > the submissions received. Sessions will be organized according=
> to
> > > > content, and not every accepted subm= ission or invited attendee
> will
> > > > have an = opportunity to present as the intent is to foster
> discussion
> > > > and not simply to have a sequence of presentations.> > > >
> > > > Position papers from those = not planning to attend the virtual
> sessions
> > > &= gt; themselves are also encouraged. A workshop report will be
> pub= lished
> > > > afterwards.
> > > >
&= gt; > > > Overview:
> > > >
> > > &= gt; "We believe that one of the major factors behind this lack of
>= progress
> > > > is the popular perception that throughpu= t is the often sole
> measure of
> > > > the quali= ty of Internet connectivity. With such narrow focus,
> people
= > > > > don=E2=80=99t consider questions such as:
> >= ; > >
> > > > What is the latency under typical work= ing conditions?
> > > > How reliable is the connectivity a= cross longer time periods?
> > > > Does the network allow = the use of a broad range of protocols?
> > > > What servic= es can be run by clients of the network?
> > > > What kind= of IPv4, NAT or IPv6 connectivity is offered, and
> are there fire= walls?
> > > > What security mechanisms are available for = local services,
> such as DNS?
> > > > To what deg= ree are the privacy, confidentiality, integrity
> and
> >= ; > > authenticity of user communications guarded?
> > >= ; >
> > > > Improving these aspects of network quality = will likely depend
> on
> > > > measurement and ex= posing metrics to all involved parties,
> including to
> &g= t; > > end users in a meaningful way. Such measurements and exposure<= br />> of the
> > > > right metrics will allow service = providers and network
> operators to
> > > > focus= on the aspects that impacts the users=E2=80=99 experience
> most a= nd at
> > > > the same time empowers users to choose the I= nternet service
> that will
> > > > give them the = best experience."
> > > >
> > > >
&g= t; > > > --
> > > > Latest Podcast:
> >= ; > >
> https://www.linkedin.com= /feed/update/urn:li:activity:6791014284936785920/
> <https://www.linkedin.com/feed/update/urn:li:activity:= 6791014284936785920/>
> > > >
> > > &= gt; Dave T=C3=A4ht CTO, TekLibre, LLC
> > > > ____________= ___________________________________
> > > > Cerowrt-devel = mailing list
> > > > Cerowrt-devel@lists.bufferbloat.net=
> <mailto:Cerowrt-devel@lists.bufferbloat.net>
> &= gt; > > https://lists.bufferbloat.net/listinfo/cerowrt-devel<= /a>
> <https://lists.bufferbloat.net/listinfo/cerowrt-de= vel>
> > > >
> >
> >
>= ; >
> > --
> > Latest Podcast:
> > https://www.linkedin.com/feed/update/urn:li:activi= ty:6791014284936785920/
> <http= s://www.linkedin.com/feed/update/urn:li:activity:6791014284936785920/&g= t;
> >
> > Dave T=C3=A4ht CTO, TekLibre, LLC
>= ; > _______________________________________________
> > Make-= wifi-fast mailing list
> > Make-wifi-fast@lists.bufferbloat.net
> <mailto:Make-wifi-fast@lists.bufferbloat.net>
&g= t; > https://lists.bufferbloat.net/listinfo/make-wifi-fast<= br />> <https://lists.bufferbloat.net/listinfo/make-wifi-fas= t>
> >
> >
> > This electronic comm= unication and the information and any files transmitted
> with it, = or attached to it, are confidential and are intended solely for the use
> of
> > the individual or entity to whom it is addressed a= nd may contain information
> that is confidential, legally privileg= ed, protected by privacy laws, or otherwise
> > restricted from = disclosure to anyone else. If you are not the intended
> recipient = or the person responsible for delivering the e-mail to the intended
&g= t; recipient,
> > you are hereby notified that any use, copying,= distributing, dissemination,
> forwarding, printing, or copying of= this e-mail is strictly prohibited. If you
> > received this e-= mail in error, please return the e-mail to the sender, delete
> it = from your computer, and destroy any printed copy of it.
> >
> > _______________________________________________
> > S= tarlink mailing list
> > Starlink@lists.bufferbloat.net
> &= gt; https://lists.bufferbloat.net/listinfo/starlink
> >>
>
> --
> Ben Greear <greearb@candelatech.com>= ;
> Candela Technologies Inc http://www.candelatech.com
>
=0A
= =0A_______________________________________________
Starlink mailing li= st
Starlink@lists.bufferbloat.net
https://lists.bufferbloat.net/l= istinfo/starlink
=0A
=0A
=0A
=0A
=0A___= ____________________________________________
Make-wifi-fast mailing l= ist
Make-wifi-fast@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/make-wifi-fast=0A
=0A
=0A
=0A=0A ------=_20210709153123000000_13915--