From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp88.iad3a.emailsrvr.com (smtp88.iad3a.emailsrvr.com [173.203.187.88]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 24B3E3B2A4 for ; Tue, 12 Dec 2017 17:53:51 -0500 (EST) Received: from smtp28.relay.iad3a.emailsrvr.com (localhost [127.0.0.1]) by smtp28.relay.iad3a.emailsrvr.com (SMTP Server) with ESMTP id D3212534A; Tue, 12 Dec 2017 17:53:50 -0500 (EST) Received: from app40.wa-webapps.iad3a (relay-webapps.rsapps.net [172.27.255.140]) by smtp28.relay.iad3a.emailsrvr.com (SMTP Server) with ESMTP id ADE07537E; Tue, 12 Dec 2017 17:53:50 -0500 (EST) X-Sender-Id: dpreed@reed.com Received: from app40.wa-webapps.iad3a (relay-webapps.rsapps.net [172.27.255.140]) by 0.0.0.0:25 (trex/5.7.12); Tue, 12 Dec 2017 17:53:50 -0500 Received: from reed.com (localhost.localdomain [127.0.0.1]) by app40.wa-webapps.iad3a (Postfix) with ESMTP id 9C84720055; Tue, 12 Dec 2017 17:53:50 -0500 (EST) Received: by apps.rackspace.com (Authenticated sender: dpreed@reed.com, from: dpreed@reed.com) with HTTP; Tue, 12 Dec 2017 17:53:50 -0500 (EST) X-Auth-ID: dpreed@reed.com Date: Tue, 12 Dec 2017 17:53:50 -0500 (EST) From: dpreed@reed.com To: "Dave Taht" Cc: "Luca Muscariello" , "Mikael Abrahamsson" , "cerowrt-devel@lists.bufferbloat.net" , "bloat" MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_20171212175350000000_92161" Importance: Normal X-Priority: 3 (Normal) X-Type: html In-Reply-To: <87wp1rbxo8.fsf@nemesis.taht.net> References: <92906bd8-7bad-945d-83c8-a2f9598aac2c@lackof.org> <87bmjff7l6.fsf_-_@nemesis.taht.net> <1512417597.091724124@apps.rackspace.com> <87wp1rbxo8.fsf@nemesis.taht.net> Message-ID: <1513119230.638732339@apps.rackspace.com> X-Mailer: webmail/12.9.10-RC Subject: Re: [Bloat] [Cerowrt-devel] DC behaviors today X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Dec 2017 22:53:51 -0000 ------=_20171212175350000000_92161 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable =0ALuca's point tends to be correct - variable latency destroys the stabili= ty of flow control loops, which destroys throughput, even when there is suf= ficient capacity to handle the load.=0A =0AThis is an indirect result of Li= ttle's Lemma (which is strictly true only for Poisson arrival, but almost a= ny arrival process will have a similar interaction between latency and thro= ughput).=0A =0AHowever, the other reason I say what I say so strongly is th= is:=0A =0ARant on.=0A =0APeak/avg. load ratio always exceeds a factor of 10= or more, IRL. Only "benchmark setups" (or hot-rod races done for academic = reasons or marketing reasons to claim some sort of "title") operate at peak= supportable load any significant part of the time.=0A =0AThe reason for th= is is not just "fat pipes are better", but because bitrate of the underlyin= g medium is an insignificant fraction of systems operational and capital ex= pense.=0A =0ASLA's are specified in "uptime" not "bits transported", and a = clogged pipe is defined as down when latency exceeds a small number.=0A =0A= Typical operating points of corporate networks where the users are happy ar= e single-digit percentage of max load.=0A =0AThis is also true of computer = buses and memory controllers and storage interfaces IRL. Again, latency is = the primary measure, and the system never focuses on operating points anywh= ere near max throughput.=0A =0ARant off.=0A=0AOn Tuesday, December 12, 2017= 1:36pm, "Dave Taht" said:=0A=0A=0A=0A> =0A> Luca Muscariel= lo writes:=0A> =0A> > I think everything is ab= out response time, even throughput.=0A> >=0A> > If we compare the time to t= ransmit a single packet from A to B, including=0A> > propagation delay, tra= nsmission delay and queuing delay,=0A> > to the time to move a much larger = amount of data from A to B we use=0A> throughput=0A> > in this second case = because it is a normalized=0A> > quantity w.r.t. response time (bytes over = delivery time). For a single=0A> > transmission we tend to use latency.=0A>= > But in the end response time is what matters.=0A> >=0A> > Also, even ins= tantaneous throughput is well defined only for a time scale=0A> which=0A> >= has to be much larger than the min RTT (propagation + transmission delays)= =0A> > Agree also that looking at video, latency and latency budgets are be= tter=0A> > quantities than throughput. At least more accurate.=0A> >=0A> > = On Fri, Dec 8, 2017 at 8:05 AM, Mikael Abrahamsson =0A> w= rote:=0A> >=0A> > On Mon, 4 Dec 2017, dpreed@reed.com wrote:=0A> >=0A> > I = suggest we stop talking about throughput, which has been the=0A> mistaken= =0A> > idea about networking for 30-40 years.=0A> >=0A> >=0A> > We need to = talk both about latency and speed. Yes, speed is talked about=0A> too=0A> >= much (relative to RTT), but it's not irrelevant.=0A> >=0A> > Speed of ligh= t in fiber means RTT is approx 1ms per 100km, so from=0A> Stockholm=0A> > t= o SFO my RTT is never going to be significantly below 85ms (8625km=0A> grea= t=0A> > circle). It's current twice that.=0A> >=0A> > So we just have to ac= cept that some services will never be deliverable=0A> > across the wider In= ternet, but have to be deployed closer to the=0A> customer=0A> > (as per yo= ur examples, some need 1ms RTT to work well), and we need=0A> lower=0A> > a= ccess latency and lower queuing delay. So yes, agreed.=0A> >=0A> > However,= I am not going to concede that speed is "mistaken idea about=0A> > network= ing". No amount of smarter queuing is going to fix the problem if=0A> I=0A>= > don't have enough throughput available to me that I need for my=0A> appl= ication.=0A> =0A> In terms of the bellcurve here, throughput has increased = much more=0A> rapidly than than latency has decreased, for most, and in an = increasing=0A> majority of human-interactive cases (like video streaming), = we often=0A> have enough throughput.=0A> =0A> And the age old argument rega= rding "just have overcapacity, always"=0A> tends to work in these cases.=0A= > =0A> I tend not to care as much about how long it takes for things that d= o=0A> not need R/T deadlines as humans and as steering wheels do.=0A> =0A> = Propigation delay, while ultimately bound by the speed of light, is also=0A= > affected by the wires wrapping indirectly around the earth - much slower= =0A> than would be possible if we worked at it:=0A> =0A> https://arxiv.org/= pdf/1505.03449.pdf=0A> =0A> Then there's inside the boxes themselves:=0A> = =0A> A lot of my struggles of late has been to get latencies and adaquate= =0A> sampling techniques down below 3ms (my previous value for starting to= =0A> reject things due to having too much noise) - and despite trying fairl= y=0A> hard, well... a process can't even sleep accurately much below 1ms, o= n=0A> bare metal linux. A dream of mine has been 8 channel high quality aud= io,=0A> with a video delay of not much more than 2.7ms for AR applications.= =0A> =0A> For comparison, an idle quad core aarch64 and dual core x86_64:= =0A> =0A> root@nanopineo2:~# irtt sleep=0A> =0A> Testing sleep accuracy...= =0A> =0A> Sleep Duration Mean Error % Error=0A> =0A> 1ns 13.353=C2=B5s 1335= 336.9=0A> =0A> 10ns 14.34=C2=B5s 143409.5=0A> =0A> 100ns 13.343=C2=B5s 1334= 3.9=0A> =0A> 1=C2=B5s 12.791=C2=B5s 1279.2=0A> =0A> 10=C2=B5s 148.661=C2=B5= s 1486.6=0A> =0A> 100=C2=B5s 150.907=C2=B5s 150.9=0A> =0A> 1ms 168.001=C2= =B5s 16.8=0A> =0A> 10ms 131.235=C2=B5s 1.3=0A> =0A> 100ms 145.611=C2=B5s 0.= 1=0A> =0A> 200ms 162.917=C2=B5s 0.1=0A> =0A> 500ms 169.885=C2=B5s 0.0=0A> = =0A> =0A> d@nemesis:~$ irtt sleep=0A> =0A> Testing sleep accuracy...=0A> = =0A> =0A> Sleep Duration Mean Error % Error=0A> =0A> 1ns 668ns 66831.9=0A> = =0A> 10ns 672ns 6723.7=0A> =0A> 100ns 557ns 557.6=0A> =0A> 1=C2=B5s 57.749= =C2=B5s 5774.9=0A> =0A> 10=C2=B5s 63.063=C2=B5s 630.6=0A> =0A> 100=C2=B5s 6= 7.737=C2=B5s 67.7=0A> =0A> 1ms 153.978=C2=B5s 15.4=0A> =0A> 10ms 169.709=C2= =B5s 1.7=0A> =0A> 100ms 186.685=C2=B5s 0.2=0A> =0A> 200ms 176.859=C2=B5s 0.= 1=0A> =0A> 500ms 177.271=C2=B5s 0.0=0A> =0A> >=0A> > --=0A> > Mikael Abraha= msson email: swmike@swm.pp.se=0A> > _______________________________________= ________=0A> >=0A> >=0A> > Bloat mailing list=0A> > Bloat@lists.bufferbloat= .net=0A> > https://lists.bufferbloat.net/listinfo/bloat=0A> >=0A> >=0A> >= =0A> > _______________________________________________=0A> > Bloat mailing = list=0A> > Bloat@lists.bufferbloat.net=0A> > https://lists.bufferbloat.net/= listinfo/bloat=0A> ------=_20171212175350000000_92161 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

Luca's point tends= to be correct - variable latency destroys the stability of flow control lo= ops, which destroys throughput, even when there is sufficient capacity to h= andle the load.

=0A

 

=0A

This is an indirect result of Little's Lemma (which is strictly true = only for Poisson arrival, but almost any arrival process will have a simila= r interaction between latency and throughput).

=0A

 

=0A

However, the other reason I say what I= say so strongly is this:

=0A

 

=0A

Rant on.

=0A

 

=0A

Peak/avg. load ratio always exceeds a factor of 10 or more, = IRL. Only "benchmark setups" (or hot-rod races done for academic reasons or= marketing reasons to claim some sort of "title") operate at peak supportab= le load any significant part of the time.

=0A

&nbs= p;

=0A

The reason for this is not just "fat pipes = are better", but because bitrate of the underlying medium is an insignifica= nt fraction of systems operational and capital expense.

=0A

 

=0A

SLA's are specified in "uptim= e" not "bits transported", and a clogged pipe is defined as down when laten= cy exceeds a small number.

=0A

 

=0A

Typical operating points of corporate networks where the u= sers are happy are single-digit percentage of max load.

=0A

 

=0A

This is also true of computer= buses and memory controllers and storage interfaces IRL. Again, latency is= the primary measure, and the system never focuses on operating points anyw= here near max throughput.

=0A

 

=0A

Rant off.

On Tuesday, December 12, 2017 1:36pm, = "Dave Taht" <dave@taht.net> said:

=0A
=0A

>
> Luca Muscariello = <luca.muscariello@gmail.com> writes:
>
> > I thin= k everything is about response time, even throughput.
> >
&= gt; > If we compare the time to transmit a single packet from A to B, in= cluding
> > propagation delay, transmission delay and queuing de= lay,
> > to the time to move a much larger amount of data from A= to B we use
> throughput
> > in this second case becaus= e it is a normalized
> > quantity w.r.t. response time (bytes ov= er delivery time). For a single
> > transmission we tend to use = latency.
> > But in the end response time is what matters.
= > >
> > Also, even instantaneous throughput is well define= d only for a time scale
> which
> > has to be much large= r than the min RTT (propagation + transmission delays)
> > Agree= also that looking at video, latency and latency budgets are better
&g= t; > quantities than throughput. At least more accurate.
> ><= br />> > On Fri, Dec 8, 2017 at 8:05 AM, Mikael Abrahamsson <swmik= e@swm.pp.se>
> wrote:
> >
> > On Mon, 4 De= c 2017, dpreed@reed.com wrote:
> >
> > I suggest we s= top talking about throughput, which has been the
> mistaken
&g= t; > idea about networking for 30-40 years.
> >
> >= ;
> > We need to talk both about latency and speed. Yes, speed i= s talked about
> too
> > much (relative to RTT), but it'= s not irrelevant.
> >
> > Speed of light in fiber mea= ns RTT is approx 1ms per 100km, so from
> Stockholm
> > = to SFO my RTT is never going to be significantly below 85ms (8625km
&g= t; great
> > circle). It's current twice that.
> >> > So we just have to accept that some services will never be del= iverable
> > across the wider Internet, but have to be deployed = closer to the
> customer
> > (as per your examples, some= need 1ms RTT to work well), and we need
> lower
> > acc= ess latency and lower queuing delay. So yes, agreed.
> >
&g= t; > However, I am not going to concede that speed is "mistaken idea abo= ut
> > networking". No amount of smarter queuing is going to fix= the problem if
> I
> > don't have enough throughput ava= ilable to me that I need for my
> application.
>
>= In terms of the bellcurve here, throughput has increased much more
&g= t; rapidly than than latency has decreased, for most, and in an increasing<= br />> majority of human-interactive cases (like video streaming), we of= ten
> have enough throughput.
>
> And the age old = argument regarding "just have overcapacity, always"
> tends to work= in these cases.
>
> I tend not to care as much about how = long it takes for things that do
> not need R/T deadlines as humans= and as steering wheels do.
>
> Propigation delay, while u= ltimately bound by the speed of light, is also
> affected by the wi= res wrapping indirectly around the earth - much slower
> than would= be possible if we worked at it:
>
> https://arxiv.org/pdf= /1505.03449.pdf
>
> Then there's inside the boxes themselv= es:
>
> A lot of my struggles of late has been to get late= ncies and adaquate
> sampling techniques down below 3ms (my previou= s value for starting to
> reject things due to having too much nois= e) - and despite trying fairly
> hard, well... a process can't even= sleep accurately much below 1ms, on
> bare metal linux. A dream of= mine has been 8 channel high quality audio,
> with a video delay o= f not much more than 2.7ms for AR applications.
>
> For co= mparison, an idle quad core aarch64 and dual core x86_64:
>
&= gt; root@nanopineo2:~# irtt sleep
>
> Testing sleep accura= cy...
>
> Sleep Duration Mean Error % Error
>
> 1ns 13.353=C2=B5s 1335336.9
>
> 10ns 14.34=C2=B5s 1= 43409.5
>
> 100ns 13.343=C2=B5s 13343.9
>
&g= t; 1=C2=B5s 12.791=C2=B5s 1279.2
>
> 10=C2=B5s 148.661=C2= =B5s 1486.6
>
> 100=C2=B5s 150.907=C2=B5s 150.9
> =
> 1ms 168.001=C2=B5s 16.8
>
> 10ms 131.235=C2=B5s= 1.3
>
> 100ms 145.611=C2=B5s 0.1
>
> 200= ms 162.917=C2=B5s 0.1
>
> 500ms 169.885=C2=B5s 0.0
&g= t;
>
> d@nemesis:~$ irtt sleep
>
> Testi= ng sleep accuracy...
>
>
> Sleep Duration Mean Er= ror % Error
>
> 1ns 668ns 66831.9
>
> 10n= s 672ns 6723.7
>
> 100ns 557ns 557.6
>
> = 1=C2=B5s 57.749=C2=B5s 5774.9
>
> 10=C2=B5s 63.063=C2=B5s = 630.6
>
> 100=C2=B5s 67.737=C2=B5s 67.7
>
&g= t; 1ms 153.978=C2=B5s 15.4
>
> 10ms 169.709=C2=B5s 1.7
>
> 100ms 186.685=C2=B5s 0.2
>
> 200ms 176.8= 59=C2=B5s 0.1
>
> 500ms 177.271=C2=B5s 0.0
>
> >
> > --
> > Mikael Abrahamsson email: swmik= e@swm.pp.se
> > _______________________________________________<= br />> >
> >
> > Bloat mailing list
> &= gt; Bloat@lists.bufferbloat.net
> > https://lists.bufferbloat.ne= t/listinfo/bloat
> >
> >
> >
> >= ; _______________________________________________
> > Bloat mail= ing list
> > Bloat@lists.bufferbloat.net
> > https://= lists.bufferbloat.net/listinfo/bloat
>

=0A
 ------=_20171212175350000000_92161--