From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from outbound01.roaringpenguin.co.uk (outbound01.roaringpenguin.co.uk [109.69.238.88]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id F0CBB3B29E; Wed, 13 Dec 2017 10:27:15 -0500 (EST) Received: from relay.smtp.pnsol.com (ec2-54-246-165-192.eu-west-1.compute.amazonaws.com [54.246.165.192]) by outbound01.roaringpenguin.co.uk (8.14.4/8.14.4/Debian-8+deb8u2) with ESMTP id vBDFR8l0015007 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA256 bits=256 verify=NOT); Wed, 13 Dec 2017 15:27:10 GMT Received: from mail.la.pnsol.com ([172.20.5.206]) by relay.smtp.pnsol.com with esmtp (Exim 4.82) (envelope-from ) id 1eP8vR-0000iT-Bn; Wed, 13 Dec 2017 15:26:25 +0000 Received: from git.pnsol.com ([172.20.5.238] helo=roam.smtp.pnsol.com) by mail.la.pnsol.com with esmtp (Exim 4.76) (envelope-from ) id 1eP8vL-0005DO-1Q; Wed, 13 Dec 2017 15:26:19 +0000 Received: from [172.20.5.109] by roam.smtp.pnsol.com with esmtpsa (TLS1.2:DHE_RSA_AES_256_CBC_SHA256:256) (Exim 4.82) (envelope-from ) id 1eP8vK-0001ud-G7; Wed, 13 Dec 2017 15:26:18 +0000 Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\)) Content-Type: multipart/signed; boundary="Apple-Mail=_305E5237-0D35-4EF0-98E6-7890867F2596"; protocol="application/pgp-signature"; micalg=pgp-sha512 From: Neil Davies X-Priority: 3 (Normal) X-Mailbutler-Link-Tracking-Uuid: In-Reply-To: <1513119230.638732339@apps.rackspace.com> Date: Wed, 13 Dec 2017 15:26:50 +0000 Cc: bloat , "cerowrt-devel@lists.bufferbloat.net" Message-Id: <7D300E07-536C-4ABD-AE38-DDBAF30E80D7@pnsol.com> References: <92906bd8-7bad-945d-83c8-a2f9598aac2c@lackof.org> <87bmjff7l6.fsf_-_@nemesis.taht.net> <1512417597.091724124@apps.rackspace.com> <87wp1rbxo8.fsf@nemesis.taht.net> <1513119230.638732339@apps.rackspace.com> To: dpreed@reed.com X-Mailer: Apple Mail (2.3273) X-Spam-Score: undef - relay 54.246.165.192 marked with skip_spam_scan X-CanIt-Geo: ip=54.246.165.192; country=IE; region=Leinster; city=Dublin; latitude=53.3389; longitude=-6.2595; http://maps.google.com/maps?q=53.3389,-6.2595&z=6 X-CanItPRO-Stream: pnsol-com:outbound (inherits from pnsol-com:default, PredictableNetworkSolutions:default, Wholesale:default, CTL:default, base:default) X-Canit-Stats-ID: Bayes signature not available X-CanIt-Archive-Cluster: Rl4fxI1RVOq/TTUYGiKHQwqlXy8 X-CanIt-Archived-As: pnsol-com/20171213 / 07UJrr8QG X-Scanned-By: CanIt (www . roaringpenguin . com) on 109.69.238.88 Subject: Re: [Cerowrt-devel] [Bloat] DC behaviors today X-BeenThere: cerowrt-devel@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Development issues regarding the cerowrt test router project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Dec 2017 15:27:16 -0000 --Apple-Mail=_305E5237-0D35-4EF0-98E6-7890867F2596 Content-Type: multipart/alternative; boundary="Apple-Mail=_08C2FE50-592D-4720-9A7D-85EE16E1F785" --Apple-Mail=_08C2FE50-592D-4720-9A7D-85EE16E1F785 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On 12 Dec 2017, at 22:53, dpreed@reed.com wrote: >=20 > Luca's point tends to be correct - variable latency destroys the = stability of flow control loops, which destroys throughput, even when = there is sufficient capacity to handle the load. >=20 > This is an indirect result of Little's Lemma (which is strictly true = only for Poisson arrival, but almost any arrival process will have a = similar interaction between latency and throughput). Actually it is true for general arrival patterns (can=E2=80=99t lay my = hands on the reference for the moment - but it was a while back that was = shown) - what this points to is an underlying conservation law - that = =E2=80=9Cdelay and loss=E2=80=9D are conserved in a scheduling process. = This comes out of the M/M/1/K/K queueing system and associated analysis. There is conservation law (and Klienrock refers to this - at least in = terms of delay - in 1965 - = http://onlinelibrary.wiley.com/doi/10.1002/nav.3800120206/abstract = ) at = work here. All scheduling systems can do is =E2=80=9Cdistribute=E2=80=9D the = resulting =E2=80=9Cdelay and loss=E2=80=9D differentially amongst the = (instantaneous set of) competing streams. Let me just repeat that - The =E2=80=9Cdelay and loss=E2=80=9D are a = conserved quantity - scheduling can=E2=80=99t =E2=80=9Cdestroy=E2=80=9D = it (they can influence higher level protocol behaviour) but not reduce = the total amount of =E2=80=9Cdelay and loss=E2=80=9D that is being = induced into the collective set of streams... >=20 > However, the other reason I say what I say so strongly is this: >=20 > Rant on. >=20 > Peak/avg. load ratio always exceeds a factor of 10 or more, IRL. Only = "benchmark setups" (or hot-rod races done for academic reasons or = marketing reasons to claim some sort of "title") operate at peak = supportable load any significant part of the time. Have you considered what this means for the economics of the operation = of networks? What other industry that =E2=80=9Cmoves things around=E2=80=9D= (i.e logistical or similar) system creates a solution in which they = have 10x as much infrastructure than their peak requirement? >=20 > The reason for this is not just "fat pipes are better", but because = bitrate of the underlying medium is an insignificant fraction of systems = operational and capital expense. Agree that (if you are the incumbent that =E2=80=98owns=E2=80=99 the low = level transmission medium) that this is true (though the costs of = lighting a new lambda are not trivial) - but that is not the experience = of anyone else in the digital supply time >=20 > SLA's are specified in "uptime" not "bits transported", and a clogged = pipe is defined as down when latency exceeds a small number. Do you have any evidence you can reference for an SLA that treats a few = ms as =E2=80=9Cdown=E2=80=9D? Most of the SLAs I=E2=80=99ve had dealings = with use averages over fairly long time periods (e.g. a month) - and = there is no quality in averages. >=20 > Typical operating points of corporate networks where the users are = happy are single-digit percentage of max load. Or less - they also detest the costs that they have to pay the network = providers to try and de-risk their applications. There is also the issue = that they measure averages (over 5min to 15min) they completely fail to = capture (for example) the 15seconds when delay and jitter was high so = the CEO=E2=80=99s video conference broke up. >=20 > This is also true of computer buses and memory controllers and storage = interfaces IRL. Again, latency is the primary measure, and the system = never focuses on operating points anywhere near max throughput. Agreed - but wouldn=E2=80=99t it be nice if they could? I=E2=80=99ve = worked on h/w systems where we have designed system to run near limits = (the set-top box market is pretty cut-throat and the closer to = saturation you can run and still deliver the acceptable outcome the = cheaper the box the greater the profit margin for the set-top box = provider) >=20 > Rant off. >=20 Cheers Neil > On Tuesday, December 12, 2017 1:36pm, "Dave Taht" > said: >=20 > > > > Luca Muscariello writes: > > > > > I think everything is about response time, even throughput. > > > > > > If we compare the time to transmit a single packet from A to B, = including > > > propagation delay, transmission delay and queuing delay, > > > to the time to move a much larger amount of data from A to B we = use > > throughput > > > in this second case because it is a normalized > > > quantity w.r.t. response time (bytes over delivery time). For a = single > > > transmission we tend to use latency. > > > But in the end response time is what matters. > > > > > > Also, even instantaneous throughput is well defined only for a = time scale > > which > > > has to be much larger than the min RTT (propagation + transmission = delays) > > > Agree also that looking at video, latency and latency budgets are = better > > > quantities than throughput. At least more accurate. > > > > > > On Fri, Dec 8, 2017 at 8:05 AM, Mikael Abrahamsson = > > wrote: > > > > > > On Mon, 4 Dec 2017, dpreed@reed.com wrote: > > > > > > I suggest we stop talking about throughput, which has been the > > mistaken > > > idea about networking for 30-40 years. > > > > > > > > > We need to talk both about latency and speed. Yes, speed is talked = about > > too > > > much (relative to RTT), but it's not irrelevant. > > > > > > Speed of light in fiber means RTT is approx 1ms per 100km, so from > > Stockholm > > > to SFO my RTT is never going to be significantly below 85ms = (8625km > > great > > > circle). It's current twice that. > > > > > > So we just have to accept that some services will never be = deliverable > > > across the wider Internet, but have to be deployed closer to the > > customer > > > (as per your examples, some need 1ms RTT to work well), and we = need > > lower > > > access latency and lower queuing delay. So yes, agreed. > > > > > > However, I am not going to concede that speed is "mistaken idea = about > > > networking". No amount of smarter queuing is going to fix the = problem if > > I > > > don't have enough throughput available to me that I need for my > > application. > > > > In terms of the bellcurve here, throughput has increased much more > > rapidly than than latency has decreased, for most, and in an = increasing > > majority of human-interactive cases (like video streaming), we often > > have enough throughput. > > > > And the age old argument regarding "just have overcapacity, always" > > tends to work in these cases. > > > > I tend not to care as much about how long it takes for things that = do > > not need R/T deadlines as humans and as steering wheels do. > > > > Propigation delay, while ultimately bound by the speed of light, is = also > > affected by the wires wrapping indirectly around the earth - much = slower > > than would be possible if we worked at it: > > > > https://arxiv.org/pdf/1505.03449.pdf > > > > Then there's inside the boxes themselves: > > > > A lot of my struggles of late has been to get latencies and adaquate > > sampling techniques down below 3ms (my previous value for starting = to > > reject things due to having too much noise) - and despite trying = fairly > > hard, well... a process can't even sleep accurately much below 1ms, = on > > bare metal linux. A dream of mine has been 8 channel high quality = audio, > > with a video delay of not much more than 2.7ms for AR applications. > > > > For comparison, an idle quad core aarch64 and dual core x86_64: > > > > root@nanopineo2:~# irtt sleep > > > > Testing sleep accuracy... > > > > Sleep Duration Mean Error % Error > > > > 1ns 13.353=C2=B5s 1335336.9 > > > > 10ns 14.34=C2=B5s 143409.5 > > > > 100ns 13.343=C2=B5s 13343.9 > > > > 1=C2=B5s 12.791=C2=B5s 1279.2 > > > > 10=C2=B5s 148.661=C2=B5s 1486.6 > > > > 100=C2=B5s 150.907=C2=B5s 150.9 > > > > 1ms 168.001=C2=B5s 16.8 > > > > 10ms 131.235=C2=B5s 1.3 > > > > 100ms 145.611=C2=B5s 0.1 > > > > 200ms 162.917=C2=B5s 0.1 > > > > 500ms 169.885=C2=B5s 0.0 > > > > > > d@nemesis:~$ irtt sleep > > > > Testing sleep accuracy... > > > > > > Sleep Duration Mean Error % Error > > > > 1ns 668ns 66831.9 > > > > 10ns 672ns 6723.7 > > > > 100ns 557ns 557.6 > > > > 1=C2=B5s 57.749=C2=B5s 5774.9 > > > > 10=C2=B5s 63.063=C2=B5s 630.6 > > > > 100=C2=B5s 67.737=C2=B5s 67.7 > > > > 1ms 153.978=C2=B5s 15.4 > > > > 10ms 169.709=C2=B5s 1.7 > > > > 100ms 186.685=C2=B5s 0.2 > > > > 200ms 176.859=C2=B5s 0.1 > > > > 500ms 177.271=C2=B5s 0.0 > > > > > > > > -- > > > Mikael Abrahamsson email: swmike@swm.pp.se > > > _______________________________________________ > > > > > > > > > Bloat mailing list > > > Bloat@lists.bufferbloat.net > > > https://lists.bufferbloat.net/listinfo/bloat > > > > > > > > > > > > _______________________________________________ > > > Bloat mailing list > > > Bloat@lists.bufferbloat.net > > > https://lists.bufferbloat.net/listinfo/bloat > > >=20 > Spam = > Not spam = > Forget previous vote = > _______________________________________________ > Bloat mailing list > Bloat@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/bloat = --Apple-Mail=_08C2FE50-592D-4720-9A7D-85EE16E1F785 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8
On 12 Dec 2017, at 22:53, dpreed@reed.com wrote:

Luca's point tends to be = correct - variable latency destroys the stability of flow control loops, = which destroys throughput, even when there is sufficient capacity to = handle the load.

 

This is an indirect result of Little's Lemma (which is = strictly true only for Poisson arrival, but almost any arrival process = will have a similar interaction between latency and = throughput).

Actually it is true for general arrival patterns = (can=E2=80=99t lay my hands on the reference for the moment - but it was = a while back that was shown) - what this points to is an underlying = conservation law - that =E2=80=9Cdelay and loss=E2=80=9D are conserved = in a scheduling process. This comes out of the M/M/1/K/K queueing system = and associated analysis.

There is =  conservation law (and Klienrock refers to this - at least in terms = of delay - in 1965 - http://onlinelibrary.wiley.com/doi/10.1002/nav.3800120206/abstr= act) at work here.

All = scheduling systems can do is =E2=80=9Cdistribute=E2=80=9D the resulting = =E2=80=9Cdelay and loss=E2=80=9D differentially amongst the = (instantaneous set of) competing streams. 

Let me just repeat that - The =E2=80=9Cdelay and = loss=E2=80=9D are a conserved quantity - scheduling can=E2=80=99t = =E2=80=9Cdestroy=E2=80=9D it (they can influence higher level protocol = behaviour) but not reduce the total amount of =E2=80=9Cdelay and loss=E2=80= =9D that is being induced into the collective set of streams...

 

However, the other reason I = say what I say so strongly is this:

 

Rant on.

 

Peak/avg. load ratio always exceeds a factor of 10 or more, = IRL. Only "benchmark setups" (or hot-rod races done for academic reasons = or marketing reasons to claim some sort of "title") operate at peak = supportable load any significant part of the = time.

Have = you considered what this means for the economics of the operation of = networks? What other industry that =E2=80=9Cmoves things around=E2=80=9D = (i.e logistical or similar) system creates a solution in which they have = 10x as much infrastructure than their peak requirement?

 

The reason for this is not = just "fat pipes are better", but because bitrate of the underlying = medium is an insignificant fraction of systems operational and capital = expense.

Agree that (if you are the incumbent that = =E2=80=98owns=E2=80=99 the low level transmission medium) that this is = true (though the costs of lighting a new lambda are not trivial) - but = that is not the experience of anyone else in the digital supply = time

 

SLA's are specified in = "uptime" not "bits transported", and a clogged pipe is defined as down = when latency exceeds a small = number.

Do = you have any evidence you can reference for an SLA that treats a few ms = as =E2=80=9Cdown=E2=80=9D? Most of the SLAs I=E2=80=99ve had dealings = with use averages over fairly long time periods (e.g. a month) - and = there is no quality in averages.

 

Typical operating points of corporate networks where the = users are happy are single-digit percentage of max = load.

Or = less - they also detest the costs that they have to pay the network = providers to try and de-risk their applications. There is also the issue = that they measure averages (over 5min to 15min) they completely fail to = capture (for example) the 15seconds when delay and jitter was high so = the CEO=E2=80=99s video conference broke up.

 

This is also true of = computer buses and memory controllers and storage interfaces IRL. Again, = latency is the primary measure, and the system never focuses on = operating points anywhere near max = throughput.

Agreed - but wouldn=E2=80=99t it be nice if they = could? I=E2=80=99ve worked on h/w systems where we have designed system = to run near limits (the set-top box market is pretty cut-throat and the = closer to saturation you can run and still deliver the acceptable = outcome the cheaper the box the greater the profit margin for the = set-top box provider)

 

Rant off.


Cheers

Neil


On = Tuesday, December 12, 2017 1:36pm, "Dave Taht" <dave@taht.net> said:

> 
> Luca = Muscariello <luca.muscariello@gmail.com> writes:
> 
> > I think everything is about response time, even = throughput.
> >
> > If we = compare the time to transmit a single packet from A to B, including
> > propagation delay, transmission delay and queuing = delay,
> > to the time to move a much larger amount = of data from A to B we use
> throughput
> > in this second case because it is a normalized
> > quantity w.r.t. response time (bytes over delivery = time). For a single
> > transmission we tend to use = latency.
> > But in the end response time is what = matters.
> >
> > Also, even = instantaneous throughput is well defined only for a time scale
> which
> > has to be much larger than = the min RTT (propagation + transmission delays)
> > = Agree also that looking at video, latency and latency budgets are = better
> > quantities than throughput. At least more = accurate.
> >
> > On Fri, Dec 8, = 2017 at 8:05 AM, Mikael Abrahamsson <swmike@swm.pp.se>
> wrote:
> >
> > = On Mon, 4 Dec 2017, dpreed@reed.com wrote:
> >
> > I suggest we stop talking about throughput, which = has been the
> mistaken
> > idea = about networking for 30-40 years.
> >
> >
> > We need to talk both about = latency and speed. Yes, speed is talked about
> too
> > much (relative to RTT), but it's not irrelevant.
> >
> > Speed of light in fiber = means RTT is approx 1ms per 100km, so from
> = Stockholm
> > to SFO my RTT is never going to be = significantly below 85ms (8625km
> great
> > circle). It's current twice that.
> = >
> > So we just have to accept that some = services will never be deliverable
> > across the = wider Internet, but have to be deployed closer to the
> = customer
> > (as per your examples, some need 1ms = RTT to work well), and we need
> lower
>= > access latency and lower queuing delay. So yes, agreed.
> >
> > However, I am not going to = concede that speed is "mistaken idea about
> > = networking". No amount of smarter queuing is going to fix the problem = if
> I
> > don't have enough = throughput available to me that I need for my
> = application.
> 
> In = terms of the bellcurve here, throughput has increased much more
> rapidly than than latency has decreased, for most, and = in an increasing
> majority of human-interactive cases = (like video streaming), we often
> have enough = throughput.
> 
> And the = age old argument regarding "just have overcapacity, always"
> tends to work in these cases.
> 
> I tend = not to care as much about how long it takes for things that do
> not need R/T deadlines as humans and as steering wheels = do.
> 
> = Propigation delay, while ultimately bound by the speed of light, is = also
> affected by the wires wrapping indirectly around = the earth - much slower
> than would be possible if we = worked at it:
> 
> https://arxiv.org/pdf/1505.03449.pdf
> 
> Then there's inside the boxes themselves:
> 
> A lot of my struggles of late has been to get latencies = and adaquate
> sampling techniques down below 3ms (my = previous value for starting to
> reject things due to = having too much noise) - and despite trying fairly
> = hard, well... a process can't even sleep accurately much below 1ms, = on
> bare metal linux. A dream of mine has been 8 = channel high quality audio,
> with a video delay of not = much more than 2.7ms for AR applications.
> 
> For = comparison, an idle quad core aarch64 and dual core x86_64:
> 
> root@nanopineo2:~# irtt sleep
> 
> Testing = sleep accuracy...
> 
> Sleep = Duration Mean Error % Error
> 
> 1ns = 13.353=C2=B5s 1335336.9
> 
> 10ns = 14.34=C2=B5s 143409.5
> 
> 100ns = 13.343=C2=B5s 13343.9
> 
> 1=C2=B5s = 12.791=C2=B5s 1279.2
> 
> 10=C2=B5s= 148.661=C2=B5s 1486.6
> 
> 100=C2=B5= s 150.907=C2=B5s 150.9
> 
> 1ms = 168.001=C2=B5s 16.8
> 
> 10ms = 131.235=C2=B5s 1.3
> 
> 100ms = 145.611=C2=B5s 0.1
> 
> 200ms = 162.917=C2=B5s 0.1
> 
> 500ms = 169.885=C2=B5s 0.0
> 
> 
> = d@nemesis:~$ irtt sleep
> 
> Testing = sleep accuracy...
> 
> 
> Sleep = Duration Mean Error % Error
> 
> 1ns = 668ns 66831.9
> 
> 10ns = 672ns 6723.7
> 
> 100ns = 557ns 557.6
> 
> 1=C2=B5s = 57.749=C2=B5s 5774.9
> 
> 10=C2=B5s= 63.063=C2=B5s 630.6
> 
> 100=C2=B5= s 67.737=C2=B5s 67.7
> 
> 1ms = 153.978=C2=B5s 15.4
> 
> 10ms = 169.709=C2=B5s 1.7
> 
> 100ms = 186.685=C2=B5s 0.2
> 
> 200ms = 176.859=C2=B5s 0.1
> 
> 500ms = 177.271=C2=B5s 0.0
> 
> >
> > --
> > Mikael Abrahamsson = email: swmike@swm.pp.se
> > = _______________________________________________
> = >
> >
> > Bloat mailing = list
> > Bloat@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/bloat
> >
> >
> >
> > _______________________________________________
> > Bloat mailing list
> > Bloat@lists.bufferbloat.net
> > https://lists.bufferbloat.net/listinfo/bloat
>

Spam
Not spam
Forget previous vote
_______________________________________________
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat

= --Apple-Mail=_08C2FE50-592D-4720-9A7D-85EE16E1F785-- --Apple-Mail=_305E5237-0D35-4EF0-98E6-7890867F2596 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP -----BEGIN PGP SIGNATURE----- iQIcBAEBCgAGBQJaMUa6AAoJEPmpMHCy/udq/AkP/j/OF/qz3GuewsFuh25akToW AhqjLSrN1AuJTIcCfp8khvoVeb+a6aAqzJsop0+V3Psh/ijCzsO5X+9WECZtX/tU TSLWqOxrxDsXChMVeitTDr6riXSSnD2aYHK7xchvYoD/dgMXi5tOhyMDc9MTSLR2 vGDAm9x7TTzR3ucE03pAF5XwXjCSHCSrn9PjVFitcm6QlItgD396GBxFeYWHMxE3 E1ZtwbpOh0iUxb8i7DgxySsy6su0VBn3t2cSybMX9a8loTTXS1oyzvNS8whbxf/C i56ggyk4UhUDNkonORwsu0Ijb0psEGHPvv1+PFiFUbgQd1FRO5giwmFM7GR2FRIX 8K7CGmHzcRgbpB6AMXT2fYuQx/2bSCB4Tjqmo7OyJBhlj9MPFAleqVmKv+SzU35O j46XzhQE1nfrExJ+d+mTobBrGSZZJduFXGffqG+LgY8alTf6VGcEig/h8bHjZB7B rXjqZcMBup6M8gLMYm/ht0jTRabV1uVS5goVm/R/U/7hNGXgwdGPVaa/GQaRCrOt zpDm1DdGBWjkbQzt6dhkMIM0Vtt6iAI7RVMRFDT3LCwcWc+onOU+lZ1VOVAh1gqI yua3WOOfyA3YinK2NmHkEcNcHyDM5wpA8+WcskoLJ+qdddqfISrdYPQy3m5u0k7H 8ATT/wZjPdS0pbMzQ3df =S/CJ -----END PGP SIGNATURE----- --Apple-Mail=_305E5237-0D35-4EF0-98E6-7890867F2596--