From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp81.iad3a.emailsrvr.com (smtp81.iad3a.emailsrvr.com [173.203.187.81]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by huchra.bufferbloat.net (Postfix) with ESMTPS id C701121F19B for ; Wed, 28 May 2014 08:52:25 -0700 (PDT) Received: from localhost (localhost.localdomain [127.0.0.1]) by smtp11.relay.iad3a.emailsrvr.com (SMTP Server) with ESMTP id A1F102802E3; Wed, 28 May 2014 11:52:24 -0400 (EDT) X-Virus-Scanned: OK Received: from app50.wa-webapps.iad3a (relay.iad3a.rsapps.net [172.27.255.110]) by smtp11.relay.iad3a.emailsrvr.com (SMTP Server) with ESMTP id 7D745280294; Wed, 28 May 2014 11:52:24 -0400 (EDT) Received: from reed.com (localhost.localdomain [127.0.0.1]) by app50.wa-webapps.iad3a (Postfix) with ESMTP id 6DD3C382DBC; Wed, 28 May 2014 11:52:24 -0400 (EDT) Received: by apps.rackspace.com (Authenticated sender: dpreed@reed.com, from: dpreed@reed.com) with HTTP; Wed, 28 May 2014 11:52:24 -0400 (EDT) Date: Wed, 28 May 2014 11:52:24 -0400 (EDT) From: dpreed@reed.com To: "David Lang" MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_20140528115224000000_35375" Importance: Normal X-Priority: 3 (Normal) X-Type: html In-Reply-To: References: <1401048053.664331760@apps.rackspace.com> <1401079740.21369945@apps.rackspace.com> <1401112879.32226492@apps.rackspace.com> Message-ID: <1401292344.44827844@apps.rackspace.com> X-Mailer: webmail7.0 Cc: "cerowrt-devel@lists.bufferbloat.net" Subject: Re: [Cerowrt-devel] Ubiquiti QOS X-BeenThere: cerowrt-devel@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: Development issues regarding the cerowrt test router project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 May 2014 15:52:26 -0000 ------=_20140528115224000000_35375 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable =0AInteresting conversation. A particular switch has no idea of the "late= ncy budget" of a particular flow - so it cannot have its *own* latency budg= et. The switch designer has no choice but to assume that his latency budg= et is near zero.=0A =0AThe number of packets that should be sustained in fl= ight to maintain maximum throughput between the source (entry) switch and d= estination (exit) switch of the flow need be no higher than=0A =0Athe flow'= s share of bandwidth of the bottleneck=0A =0Amultiplied by=0A =0Athe end-to= -end delay (including packet forwarding, but not queueing).=0A =0AAll buffe= ring needed for isochrony ("jitter buffer") and "alternative path selection= " can be moved to either before the entry switch or after the exit switch.= =0A =0AIf you have multiple simultaneous paths, the number of packets in fl= ight involves replacing "bandwidth of the bottleneck" with "aggregate bandw= idth across the minimum cut-set of the chosen paths used for the flow".=0A = =0AOf course, these are dynamic - "the flow's share" and "paths used for th= e flow" change over short time scales. That's why you have a control loop = that needs to measure them.=0A =0AThe whole point of minimizing buffering i= s to make the measurements more timely and the control inputs more timely. = This is not about convergence to an asymptote....=0A =0AA network where ev= ery internal buffer is driven hard toward zero makes it possible to handle = multiple paths, alternate paths, etc. more *easily*. That's partly becaus= e you allow endpoints to see what is happening to their flows more quickly = so they can compensate.=0A =0AAnd of course for shared wireless resources, = things change more quickly because of new factors - more sharing, more comp= etition for collision-free slots, varying transmission rates, etc.=0A =0ATh= e last thing you want is long-term standing waves caused by large buffers a= nd very loose control.=0A =0A=0A=0AOn Tuesday, May 27, 2014 11:21pm, "David= Lang" said:=0A=0A=0A=0A> On Tue, 27 May 2014, Dave Taht wr= ote:=0A> =0A> > On Tue, May 27, 2014 at 4:27 PM, David Lang = wrote:=0A> >> On Tue, 27 May 2014, Dave Taht wrote:=0A> >>=0A> >>> There i= s a phrase in this thread that is begging to bother me.=0A> >>>=0A> >>> "Th= roughput". Everyone assumes that throughput is a big goal - and=0A> it=0A> = >>> certainly is - and latency is also a big goal - and it certainly is=0A>= -=0A> >>> but by specifying what you want from "throughput" as a compromis= e=0A> with=0A> >>> latency is not the right thing...=0A> >>>=0A> >>> If wha= t you want is actually "high speed in-order packet delivery" -=0A> >>> say,= for example a movie,=0A> >>> or a video conference, youtube, or a video co= nference - excessive=0A> >>> latency with high throughput, really, really m= akes in-order packet=0A> >>> delivery at high speed tough.=0A> >>=0A> >>=0A= > >> the key word here is "excessive", that's why I said that for max=0A> t= hroughput=0A> >> you want to buffer as much as your latency budget will all= ow you to.=0A> >=0A> > Again I'm trying to make a distinction between "thro= ughput", and "packets=0A> > delivered-in-order-to-the-user." (for-which-we-= need-a-new-word-I think)=0A> >=0A> > The buffering should not be in-the-net= work, it can be in the application.=0A> >=0A> > Take our hypothetical video= stream for example. I am 20ms RTT from netflix.=0A> > If I artificially in= flate that by adding 50ms of in-network buffering,=0A> > that means a loss = can=0A> > take 120ms to recover from.=0A> >=0A> > If instead, I keep a 3*RT= T buffer in my application, and expect that I have=0A> 5ms=0A> > worth of n= etwork-buffering, instead, I recover from a loss in 40ms.=0A> >=0A> > (plea= se note, it's late, I might not have got the math entirely right)=0A> =0A> = but you aren't going to be tuning the retry wait time per connection. what = is=0A> the retry time that is set in your stack? It's something huge to sur= vive=0A> international connections with satellite paths (so several seconds= worth). If=0A> your server-to-eyeball buffering is shorter than this, you = will get a window=0A> where you aren't fully utilizing the connection.=0A> = =0A> so yes, I do think that if your purpose is to get the maximum possible= in-order=0A> packets delivered, you end up making different decisions than= if you are just=0A> trying to stream a HD video, or do other normal things= .=0A> =0A> The problem is thinking that this absolute throughput is represe= ntitive of=0A> normal use.=0A> =0A> > As physical RTTs grow shorter, the ad= vantages of smaller buffers grow=0A> larger.=0A> >=0A> > You don't need 50m= s queueing delay on a 100us path.=0A> >=0A> > Many applications buffer for = seconds due to needing to be at least=0A> > 2*(actual buffering+RTT) on the= path.=0A> =0A> For something like streaming video, there's nothing wrong w= ith the application=0A> buffering aggressivly (assuming you have the space = to do so on the client side),=0A> the more you have gotten transmitted to t= he client, the longer it can survive a=0A> disruption of it's network.=0A> = =0A> There's nothing wrong with having an hour of buffered data between the= server=0A> and the viewer's eyes.now, this buffering should not be in the = network devices, it=0A> should be in the=0A> client app, but this isn't bec= ause there's something wrong with bufferng, it's=0A> just because the clien= t device has so much more available space to hold stuff.=0A> =0A> David Lan= g=0A> =0A> >>=0A> >>> You eventually lose a packet, and you have to wait a = really long=0A> time=0A> >>> until a replacement arrives. Stuart and I show= ed that at last ietf.=0A> >>> And you get the classic "buffering" song play= ing....=0A> >>=0A> >>=0A> >> Yep, and if you buffer too much, your "lost pa= cket" is actually still in=0A> >> flight and eating bandwidth.=0A> >>=0A> >= > David Lang=0A> >>=0A> >>=0A> >>> low latency makes recovery from a loss i= n an in-order stream much,=0A> much=0A> >>> faster.=0A> >>>=0A> >>> Honestl= y, for most applications on the web, what you want is high=0A> >>> speed in= -order packet delivery, not=0A> >>> "bulk throughput". There is a whole cla= ss of apps (bittorrent, file=0A> >>> transfer) that don't need that, and we= =0A> >>> have protocols for those....=0A> >>>=0A> >>>=0A> >>>=0A> >>> On Tu= e, May 27, 2014 at 2:19 PM, David Lang =0A> wrote:=0A> >>>>= =0A> >>>> the problem is that paths change, they mix traffic from streams,= =0A> and in=0A> >>>> other ways the utilization of the links can change rad= ically in a=0A> short=0A> >>>> amount of time.=0A> >>>>=0A> >>>> If you try= to limit things to exactly the ballistic throughput,=0A> you are=0A> >>>> = not=0A> >>>> going to be able to exactly maintain this state, you are eithe= r=0A> going to=0A> >>>> overshoot (too much traffic, requiring dropping pac= kets to=0A> maintain your=0A> >>>> minimal buffer), or you are going to und= ershoot (too little=0A> traffic and=0A> >>>> your=0A> >>>> connection is id= le)=0A> >>>>=0A> >>>> Since you can't predict all the competing traffic thr= oughout the=0A> >>>> Internet,=0A> >>>> if you want to maximize throughput,= you want to buffer as much as=0A> you can=0A> >>>> tolerate for latency re= asons. For most apps, this is more than=0A> enough to=0A> >>>> cause proble= ms for other connections.=0A> >>>>=0A> >>>> David Lang=0A> >>>>=0A> >>>>=0A= > >>>> On Mon, 26 May 2014, David P. Reed wrote:=0A> >>>>=0A> >>>>> Codel = and PIE are excellent first steps... but I don't think=0A> they are=0A> >>>= >> the=0A> >>>>> best eventual approach. I want to see them deployed ASAP = in=0A> CMTS' s and=0A> >>>>> server load balancing networks... it would be = a disaster to=0A> not deploy=0A> >>>>> the=0A> >>>>> far better option we h= ave today immediately at the point of=0A> most=0A> >>>>> leverage.=0A> >>>>= > The best is the enemy of the good.=0A> >>>>>=0A> >>>>> But, the community= needs to learn once and for all that=0A> throughput and=0A> >>>>> latency = do not trade off. We can in principle get far better=0A> latency=0A> >>>>> = while=0A> >>>>> maintaining high throughput.... and we need to start thinki= ng=0A> about=0A> >>>>> that.=0A> >>>>> That means that the framing of the i= ssue as AQM is=0A> counterproductive.=0A> >>>>>=0A> >>>>> On May 26, 2014, = Mikael Abrahamsson =0A> wrote:=0A> >>>>>>=0A> >>>>>>=0A> = >>>>>> On Mon, 26 May 2014, dpreed@reed.com wrote:=0A> >>>>>>=0A> >>>>>>> I= would look to queue minimization rather than "queue=0A> management"=0A> >>= >>>>=0A> >>>>>>=0A> >>>>>> (which=0A> >>>>>>>=0A> >>>>>>>=0A> >>>>>>> impli= ed queues are often long) as a goal, and think=0A> harder about the=0A> >>>= >>>> end-to-end problem of minimizing total end-to-end=0A> queueing delay= =0A> >>>>>>=0A> >>>>>>=0A> >>>>>> while=0A> >>>>>>>=0A> >>>>>>>=0A> >>>>>>>= maximizing throughput.=0A> >>>>>>=0A> >>>>>>=0A> >>>>>>=0A> >>>>>> As far = as I can tell, this is exactly what CODEL and PIE=0A> tries to do.=0A> >>>>= >> They=0A> >>>>>> try to find a decent tradeoff between having queues to= =0A> make sure the=0A> >>>>>> pipe=0A> >>>>>> is filled, and not making the= se queues big enough to=0A> seriously affect=0A> >>>>>> interactive perform= ance.=0A> >>>>>>=0A> >>>>>> The latter part looks like what LEDBAT does?=0A= > >>>>>> =0A> >>>>>>=0A> >>>>>> Or are = you thinking about something else?=0A> >>>>>=0A> >>>>>=0A> >>>>>=0A> >>>>> = -- Sent from my Android device with K-@ Mail. Please excuse=0A> my brevity.= =0A> >>>>=0A> >>>>=0A> >>>>=0A> >>>> ______________________________________= _________=0A> >>>> Cerowrt-devel mailing list=0A> >>>> Cerowrt-devel@lists.= bufferbloat.net=0A> >>>> https://lists.bufferbloat.net/listinfo/cerowrt-dev= el=0A> >>>>=0A> >>>> _______________________________________________=0A> >>= >> Cerowrt-devel mailing list=0A> >>>> Cerowrt-devel@lists.bufferbloat.net= =0A> >>>> https://lists.bufferbloat.net/listinfo/cerowrt-devel=0A> >>>>=0A>= >>>=0A> >>>=0A> >>>=0A> >>>=0A> >>=0A> >=0A> >=0A> >=0A> >=0A> ------=_20140528115224000000_35375 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

Interestin= g conversation.   A particular switch has no idea of the "latency budg= et" of a particular flow - so it cannot have its *own* latency budget. &nbs= p; The switch designer has no choice but to assume that his latency budget = is near zero.

=0A

 

=0A

The number of packets that should be sustained in = flight to maintain maximum throughput between the source (entry) switch and= destination (exit) switch of the flow need be no higher than

=0A

 

=0A

the= flow's share of bandwidth of the bottleneck

=0A

 

=0A

multiplied by

=0A=

 

=0A

the end-to-end delay (including packet forwarding, but not queueing).=0A

 

=0A

All buffering needed for isochrony ("jitter buffer") and "alternat= ive path selection" can be moved to either before the entry switch or after= the exit switch.

=0A

 

=0A

If you have multiple simultaneous paths, the nu= mber of packets in flight involves replacing "bandwidth of the bottleneck" = with "aggregate bandwidth across the minimum cut-set of the chosen paths us= ed for the flow".

=0A

 

=0A

Of course, these are dynamic - "the flow's shar= e" and "paths used for the flow" change over short time scales.  That'= s why you have a control loop that needs to measure them.

=0A

 

=0A

The who= le point of minimizing buffering is to make the measurements more timely an= d the control inputs more timely.  This is not about convergence to an= asymptote....

=0A

 

=0A

A network where every internal buffer is driven ha= rd toward zero makes it possible to handle multiple paths, alternate paths,= etc. more *easily*.   That's partly because you allow endpoints to se= e what is happening to their flows more quickly so they can compensate.

= =0A

 

=0A

And of course for shared wireless resources, things change more quic= kly because of new factors - more sharing, more competition for collision-f= ree slots, varying transmission rates, etc.

=0A

 

=0A

The last thing you wa= nt is long-term standing waves caused by large buffers and very loose contr= ol.

=0A

 

=0A





On Tuesday, May 27, 2014 11:21pm, "David= Lang" <david@lang.hm> said:

=0A
=0A

> On Tue, 27 May 2014, Da= ve Taht wrote:
>
> > On Tue, May 27, 2014 at 4:27 PM, D= avid Lang <david@lang.hm> wrote:
> >> On Tue, 27 May 20= 14, Dave Taht wrote:
> >>
> >>> There is a p= hrase in this thread that is begging to bother me.
> >>>> >>> "Throughput". Everyone assumes that throughput is a b= ig goal - and
> it
> >>> certainly is - and latenc= y is also a big goal - and it certainly is
> -
> >>&g= t; but by specifying what you want from "throughput" as a compromise
&= gt; with
> >>> latency is not the right thing...
>= >>>
> >>> If what you want is actually "high spe= ed in-order packet delivery" -
> >>> say, for example a mo= vie,
> >>> or a video conference, youtube, or a video conf= erence - excessive
> >>> latency with high throughput, rea= lly, really makes in-order packet
> >>> delivery at high s= peed tough.
> >>
> >>
> >> the ke= y word here is "excessive", that's why I said that for max
> throug= hput
> >> you want to buffer as much as your latency budget w= ill allow you to.
> >
> > Again I'm trying to make a = distinction between "throughput", and "packets
> > delivered-in-= order-to-the-user." (for-which-we-need-a-new-word-I think)
> >> > The buffering should not be in-the-network, it can be in the = application.
> >
> > Take our hypothetical video stre= am for example. I am 20ms RTT from netflix.
> > If I artificiall= y inflate that by adding 50ms of in-network buffering,
> > that = means a loss can
> > take 120ms to recover from.
> ><= br />> > If instead, I keep a 3*RTT buffer in my application, and exp= ect that I have
> 5ms
> > worth of network-buffering, in= stead, I recover from a loss in 40ms.
> >
> > (please= note, it's late, I might not have got the math entirely right)
> <= br />> but you aren't going to be tuning the retry wait time per connect= ion. what is
> the retry time that is set in your stack? It's somet= hing huge to survive
> international connections with satellite pat= hs (so several seconds worth). If
> your server-to-eyeball bufferin= g is shorter than this, you will get a window
> where you aren't fu= lly utilizing the connection.
>
> so yes, I do think that = if your purpose is to get the maximum possible in-order
> packets d= elivered, you end up making different decisions than if you are just
&= gt; trying to stream a HD video, or do other normal things.
>
> The problem is thinking that this absolute throughput is representiti= ve of
> normal use.
>
> > As physical RTTs grow= shorter, the advantages of smaller buffers grow
> larger.
>= ; >
> > You don't need 50ms queueing delay on a 100us path.> >
> > Many applications buffer for seconds due to ne= eding to be at least
> > 2*(actual buffering+RTT) on the path.>
> For something like streaming video, there's nothing wro= ng with the application
> buffering aggressivly (assuming you have = the space to do so on the client side),
> the more you have gotten = transmitted to the client, the longer it can survive a
> disruption= of it's network.
>
> There's nothing wrong with having an= hour of buffered data between the server
> and the viewer's eyes.n= ow, this buffering should not be in the network devices, it
> shoul= d be in the
> client app, but this isn't because there's something = wrong with bufferng, it's
> just because the client device has so m= uch more available space to hold stuff.
>
> David Lang
>
> >>
> >>> You eventually lose a pac= ket, and you have to wait a really long
> time
> >>&g= t; until a replacement arrives. Stuart and I showed that at last ietf.
> >>> And you get the classic "buffering" song playing....
> >>
> >>
> >> Yep, and if you buffe= r too much, your "lost packet" is actually still in
> >> flig= ht and eating bandwidth.
> >>
> >> David Lang> >>
> >>
> >>> low latency mak= es recovery from a loss in an in-order stream much,
> much
>= ; >>> faster.
> >>>
> >>> Honest= ly, for most applications on the web, what you want is high
> >&= gt;> speed in-order packet delivery, not
> >>> "bulk th= roughput". There is a whole class of apps (bittorrent, file
> >&= gt;> transfer) that don't need that, and we
> >>> have = protocols for those....
> >>>
> >>>
= > >>>
> >>> On Tue, May 27, 2014 at 2:19 PM, D= avid Lang <david@lang.hm>
> wrote:
> >>>>=
> >>>> the problem is that paths change, they mix traf= fic from streams,
> and in
> >>>> other ways th= e utilization of the links can change radically in a
> short
&= gt; >>>> amount of time.
> >>>>
> &= gt;>>> If you try to limit things to exactly the ballistic through= put,
> you are
> >>>> not
> >>>= ;> going to be able to exactly maintain this state, you are either
= > going to
> >>>> overshoot (too much traffic, requi= ring dropping packets to
> maintain your
> >>>>= minimal buffer), or you are going to undershoot (too little
> traf= fic and
> >>>> your
> >>>> connecti= on is idle)
> >>>>
> >>>> Since you= can't predict all the competing traffic throughout the
> >>&= gt;> Internet,
> >>>> if you want to maximize throug= hput, you want to buffer as much as
> you can
> >>>= ;> tolerate for latency reasons. For most apps, this is more than
&= gt; enough to
> >>>> cause problems for other connectio= ns.
> >>>>
> >>>> David Lang
&= gt; >>>>
> >>>>
> >>>> = On Mon, 26 May 2014, David P. Reed wrote:
> >>>>
= > >>>>> Codel and PIE are excellent first steps... but I = don't think
> they are
> >>>>> the
>= >>>>> best eventual approach. I want to see them deployed = ASAP in
> CMTS' s and
> >>>>> server load ba= lancing networks... it would be a disaster to
> not deploy
>= ; >>>>> the
> >>>>> far better option= we have today immediately at the point of
> most
> >>= ;>>> leverage.
> >>>>> The best is the enem= y of the good.
> >>>>>
> >>>>>= ; But, the community needs to learn once and for all that
> through= put and
> >>>>> latency do not trade off. We can in = principle get far better
> latency
> >>>>> w= hile
> >>>>> maintaining high throughput.... and we = need to start thinking
> about
> >>>>> that.=
> >>>>> That means that the framing of the issue as= AQM is
> counterproductive.
> >>>>>
&g= t; >>>>> On May 26, 2014, Mikael Abrahamsson <swmike@swm.= pp.se>
> wrote:
> >>>>>>
> >= ;>>>>>
> >>>>>> On Mon, 26 May 201= 4, dpreed@reed.com wrote:
> >>>>>>
> >= >>>>>> I would look to queue minimization rather than "qu= eue
> management"
> >>>>>>
> >= >>>>>
> >>>>>> (which
> &g= t;>>>>>>
> >>>>>>>
>= >>>>>>> implied queues are often long) as a goal, and= think
> harder about the
> >>>>>>> en= d-to-end problem of minimizing total end-to-end
> queueing delay> >>>>>>
> >>>>>>
&g= t; >>>>>> while
> >>>>>>>> >>>>>>>
> >>>>>>>= maximizing throughput.
> >>>>>>
> >&g= t;>>>>
> >>>>>>
> >>>= ;>>> As far as I can tell, this is exactly what CODEL and PIE
> tries to do.
> >>>>>> They
> >&g= t;>>>> try to find a decent tradeoff between having queues to> make sure the
> >>>>>> pipe
> &g= t;>>>>> is filled, and not making these queues big enough to=
> seriously affect
> >>>>>> interactive = performance.
> >>>>>>
> >>>>&= gt;> The latter part looks like what LEDBAT does?
> >>>= >>> <http://tools.ietf.org/html/rfc6817>
> >>&= gt;>>>
> >>>>>> Or are you thinking abou= t something else?
> >>>>>
> >>>>= >
> >>>>>
> >>>>> -- Sent = from my Android device with K-@ Mail. Please excuse
> my brevity.> >>>>
> >>>>
> >>>= >
> >>>> ___________________________________________= ____
> >>>> Cerowrt-devel mailing list
> >&g= t;>> Cerowrt-devel@lists.bufferbloat.net
> >>>> h= ttps://lists.bufferbloat.net/listinfo/cerowrt-devel
> >>>&= gt;
> >>>> ____________________________________________= ___
> >>>> Cerowrt-devel mailing list
> >>= ;>> Cerowrt-devel@lists.bufferbloat.net
> >>>> ht= tps://lists.bufferbloat.net/listinfo/cerowrt-devel
> >>>&g= t;
> >>>
> >>>
> >>>
> >>>
> >>
> >
> >
&= gt; >
> >
>

=0A
 ------=_20140528115224000000_35375--