<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><br class=""><div><blockquote type="cite" class=""><div class="">On 12 Dec 2017, at 22:53, <a href="mailto:dpreed@reed.com" class="">dpreed@reed.com</a> wrote:</div><br class="Apple-interchange-newline"><div class=""><font face="verdana" size="2" style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><div style="margin: 0px; padding: 0px; font-family: verdana; font-size: 10pt; overflow-wrap: break-word;" class="">Luca's point tends to be correct - variable latency destroys the stability of flow control loops, which destroys throughput, even when there is sufficient capacity to handle the load.</div><p style="margin: 0px; padding: 0px; font-family: verdana; font-size: 10pt; overflow-wrap: break-word;" class=""> </p><div style="margin: 0px; padding: 0px; font-family: verdana; font-size: 10pt; overflow-wrap: break-word;" class="">This is an indirect result of Little's Lemma (which is strictly true only for Poisson arrival, but almost any arrival process will have a similar interaction between latency and throughput).</div></font></div></blockquote><div><br class=""></div><div>Actually it is true for general arrival patterns (can’t lay my hands on the reference for the moment - but it was a while back that was shown) - what this points to is an underlying conservation law - that “delay and loss” are conserved in a scheduling process. This comes out of the M/M/1/K/K queueing system and associated analysis.</div><div><br class=""></div><div>There is conservation law (and Klienrock refers to this - at least in terms of delay - in 1965 - <a href="http://onlinelibrary.wiley.com/doi/10.1002/nav.3800120206/abstract" class="">http://onlinelibrary.wiley.com/doi/10.1002/nav.3800120206/abstract</a>) at work here.</div><div><br class=""></div><div>All scheduling systems can do is “distribute” the resulting “delay and loss” differentially amongst the (instantaneous set of) competing streams. </div><div><br class=""></div><div>Let me just repeat that - The “delay and loss” are a conserved quantity - scheduling can’t “destroy” it (they can influence higher level protocol behaviour) but not reduce the total amount of “delay and loss” that is being induced into the collective set of streams...</div><br class=""><blockquote type="cite" class=""><div class=""><font face="verdana" size="2" style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><p style="margin: 0px; padding: 0px; font-family: verdana; font-size: 10pt; overflow-wrap: break-word;" class=""> </p><div style="margin: 0px; padding: 0px; font-family: verdana; font-size: 10pt; overflow-wrap: break-word;" class="">However, the other reason I say what I say so strongly is this:</div><p style="margin: 0px; padding: 0px; font-family: verdana; font-size: 10pt; overflow-wrap: break-word;" class=""> </p><div style="margin: 0px; padding: 0px; font-family: verdana; font-size: 10pt; overflow-wrap: break-word;" class="">Rant on.</div><p style="margin: 0px; padding: 0px; font-family: verdana; font-size: 10pt; overflow-wrap: break-word;" class=""> </p><div style="margin: 0px; padding: 0px; font-family: verdana; font-size: 10pt; overflow-wrap: break-word;" class="">Peak/avg. load ratio always exceeds a factor of 10 or more, IRL. Only "benchmark setups" (or hot-rod races done for academic reasons or marketing reasons to claim some sort of "title") operate at peak supportable load any significant part of the time.</div></font></div></blockquote><div><br class=""></div><div>Have you considered what this means for the economics of the operation of networks? What other industry that “moves things around” (i.e logistical or similar) system creates a solution in which they have 10x as much infrastructure than their peak requirement?</div><br class=""><blockquote type="cite" class=""><div class=""><font face="verdana" size="2" style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><p style="margin: 0px; padding: 0px; font-family: verdana; font-size: 10pt; overflow-wrap: break-word;" class=""> </p><div style="margin: 0px; padding: 0px; font-family: verdana; font-size: 10pt; overflow-wrap: break-word;" class="">The reason for this is not just "fat pipes are better", but because bitrate of the underlying medium is an insignificant fraction of systems operational and capital expense.</div></font></div></blockquote><div><br class=""></div><div>Agree that (if you are the incumbent that ‘owns’ the low level transmission medium) that this is true (though the costs of lighting a new lambda are not trivial) - but that is not the experience of anyone else in the digital supply time</div><br class=""><blockquote type="cite" class=""><div class=""><font face="verdana" size="2" style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><p style="margin: 0px; padding: 0px; font-family: verdana; font-size: 10pt; overflow-wrap: break-word;" class=""> </p><div style="margin: 0px; padding: 0px; font-family: verdana; font-size: 10pt; overflow-wrap: break-word;" class="">SLA's are specified in "uptime" not "bits transported", and a clogged pipe is defined as down when latency exceeds a small number.</div></font></div></blockquote><div><br class=""></div><div>Do you have any evidence you can reference for an SLA that treats a few ms as “down”? Most of the SLAs I’ve had dealings with use averages over fairly long time periods (e.g. a month) - and there is no quality in averages.</div><br class=""><blockquote type="cite" class=""><div class=""><font face="verdana" size="2" style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><p style="margin: 0px; padding: 0px; font-family: verdana; font-size: 10pt; overflow-wrap: break-word;" class=""> </p><div style="margin: 0px; padding: 0px; font-family: verdana; font-size: 10pt; overflow-wrap: break-word;" class="">Typical operating points of corporate networks where the users are happy are single-digit percentage of max load.</div></font></div></blockquote><div><br class=""></div><div>Or less - they also detest the costs that they have to pay the network providers to try and de-risk their applications. There is also the issue that they measure averages (over 5min to 15min) they completely fail to capture (for example) the 15seconds when delay and jitter was high so the CEO’s video conference broke up.</div><br class=""><blockquote type="cite" class=""><div class=""><font face="verdana" size="2" style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><p style="margin: 0px; padding: 0px; font-family: verdana; font-size: 10pt; overflow-wrap: break-word;" class=""> </p><div style="margin: 0px; padding: 0px; font-family: verdana; font-size: 10pt; overflow-wrap: break-word;" class="">This is also true of computer buses and memory controllers and storage interfaces IRL. Again, latency is the primary measure, and the system never focuses on operating points anywhere near max throughput.</div></font></div></blockquote><div><br class=""></div><div>Agreed - but wouldn’t it be nice if they could? I’ve worked on h/w systems where we have designed system to run near limits (the set-top box market is pretty cut-throat and the closer to saturation you can run and still deliver the acceptable outcome the cheaper the box the greater the profit margin for the set-top box provider)</div><br class=""><blockquote type="cite" class=""><div class=""><font face="verdana" size="2" style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><p style="margin: 0px; padding: 0px; font-family: verdana; font-size: 10pt; overflow-wrap: break-word;" class=""> </p><div style="margin: 0px; padding: 0px; font-family: verdana; font-size: 10pt; overflow-wrap: break-word;" class="">Rant off.<br class=""><br class=""></div></font></div></blockquote><div><br class=""></div><div>Cheers</div><div><br class=""></div><div>Neil</div><div><br class=""></div><br class=""><blockquote type="cite" class=""><div class=""><font face="verdana" size="2" style="font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><div style="margin: 0px; padding: 0px; font-family: verdana; font-size: 10pt; overflow-wrap: break-word;" class="">On Tuesday, December 12, 2017 1:36pm, "Dave Taht" <<a href="mailto:dave@taht.net" class="">dave@taht.net</a>> said:<br class=""><br class=""></div><div id="SafeStyles1513118484" class=""><div style="margin: 0px; padding: 0px; font-family: verdana; font-size: 10pt; overflow-wrap: break-word;" class="">><span class="Apple-converted-space"> </span><br class="">> Luca Muscariello <<a href="mailto:luca.muscariello@gmail.com" class="">luca.muscariello@gmail.com</a>> writes:<br class="">><span class="Apple-converted-space"> </span><br class="">> > I think everything is about response time, even throughput.<br class="">> ><br class="">> > If we compare the time to transmit a single packet from A to B, including<br class="">> > propagation delay, transmission delay and queuing delay,<br class="">> > to the time to move a much larger amount of data from A to B we use<br class="">> throughput<br class="">> > in this second case because it is a normalized<br class="">> > quantity w.r.t. response time (bytes over delivery time). For a single<br class="">> > transmission we tend to use latency.<br class="">> > But in the end response time is what matters.<br class="">> ><br class="">> > Also, even instantaneous throughput is well defined only for a time scale<br class="">> which<br class="">> > has to be much larger than the min RTT (propagation + transmission delays)<br class="">> > Agree also that looking at video, latency and latency budgets are better<br class="">> > quantities than throughput. At least more accurate.<br class="">> ><br class="">> > On Fri, Dec 8, 2017 at 8:05 AM, Mikael Abrahamsson <<a href="mailto:swmike@swm.pp.se" class="">swmike@swm.pp.se</a>><br class="">> wrote:<br class="">> ><br class="">> > On Mon, 4 Dec 2017, <a href="mailto:dpreed@reed.com" class="">dpreed@reed.com</a> wrote:<br class="">> ><br class="">> > I suggest we stop talking about throughput, which has been the<br class="">> mistaken<br class="">> > idea about networking for 30-40 years.<br class="">> ><br class="">> ><br class="">> > We need to talk both about latency and speed. Yes, speed is talked about<br class="">> too<br class="">> > much (relative to RTT), but it's not irrelevant.<br class="">> ><br class="">> > Speed of light in fiber means RTT is approx 1ms per 100km, so from<br class="">> Stockholm<br class="">> > to SFO my RTT is never going to be significantly below 85ms (8625km<br class="">> great<br class="">> > circle). It's current twice that.<br class="">> ><br class="">> > So we just have to accept that some services will never be deliverable<br class="">> > across the wider Internet, but have to be deployed closer to the<br class="">> customer<br class="">> > (as per your examples, some need 1ms RTT to work well), and we need<br class="">> lower<br class="">> > access latency and lower queuing delay. So yes, agreed.<br class="">> ><br class="">> > However, I am not going to concede that speed is "mistaken idea about<br class="">> > networking". No amount of smarter queuing is going to fix the problem if<br class="">> I<br class="">> > don't have enough throughput available to me that I need for my<br class="">> application.<br class="">><span class="Apple-converted-space"> </span><br class="">> In terms of the bellcurve here, throughput has increased much more<br class="">> rapidly than than latency has decreased, for most, and in an increasing<br class="">> majority of human-interactive cases (like video streaming), we often<br class="">> have enough throughput.<br class="">><span class="Apple-converted-space"> </span><br class="">> And the age old argument regarding "just have overcapacity, always"<br class="">> tends to work in these cases.<br class="">><span class="Apple-converted-space"> </span><br class="">> I tend not to care as much about how long it takes for things that do<br class="">> not need R/T deadlines as humans and as steering wheels do.<br class="">><span class="Apple-converted-space"> </span><br class="">> Propigation delay, while ultimately bound by the speed of light, is also<br class="">> affected by the wires wrapping indirectly around the earth - much slower<br class="">> than would be possible if we worked at it:<br class="">><span class="Apple-converted-space"> </span><br class="">> <a href="https://arxiv.org/pdf/1505.03449.pdf" class="">https://arxiv.org/pdf/1505.03449.pdf</a><br class="">><span class="Apple-converted-space"> </span><br class="">> Then there's inside the boxes themselves:<br class="">><span class="Apple-converted-space"> </span><br class="">> A lot of my struggles of late has been to get latencies and adaquate<br class="">> sampling techniques down below 3ms (my previous value for starting to<br class="">> reject things due to having too much noise) - and despite trying fairly<br class="">> hard, well... a process can't even sleep accurately much below 1ms, on<br class="">> bare metal linux. A dream of mine has been 8 channel high quality audio,<br class="">> with a video delay of not much more than 2.7ms for AR applications.<br class="">><span class="Apple-converted-space"> </span><br class="">> For comparison, an idle quad core aarch64 and dual core x86_64:<br class="">><span class="Apple-converted-space"> </span><br class="">> root@nanopineo2:~# irtt sleep<br class="">><span class="Apple-converted-space"> </span><br class="">> Testing sleep accuracy...<br class="">><span class="Apple-converted-space"> </span><br class="">> Sleep Duration Mean Error % Error<br class="">><span class="Apple-converted-space"> </span><br class="">> 1ns 13.353µs 1335336.9<br class="">><span class="Apple-converted-space"> </span><br class="">> 10ns 14.34µs 143409.5<br class="">><span class="Apple-converted-space"> </span><br class="">> 100ns 13.343µs 13343.9<br class="">><span class="Apple-converted-space"> </span><br class="">> 1µs 12.791µs 1279.2<br class="">><span class="Apple-converted-space"> </span><br class="">> 10µs 148.661µs 1486.6<br class="">><span class="Apple-converted-space"> </span><br class="">> 100µs 150.907µs 150.9<br class="">><span class="Apple-converted-space"> </span><br class="">> 1ms 168.001µs 16.8<br class="">><span class="Apple-converted-space"> </span><br class="">> 10ms 131.235µs 1.3<br class="">><span class="Apple-converted-space"> </span><br class="">> 100ms 145.611µs 0.1<br class="">><span class="Apple-converted-space"> </span><br class="">> 200ms 162.917µs 0.1<br class="">><span class="Apple-converted-space"> </span><br class="">> 500ms 169.885µs 0.0<br class="">><span class="Apple-converted-space"> </span><br class="">><span class="Apple-converted-space"> </span><br class="">> d@nemesis:~$ irtt sleep<br class="">><span class="Apple-converted-space"> </span><br class="">> Testing sleep accuracy...<br class="">><span class="Apple-converted-space"> </span><br class="">><span class="Apple-converted-space"> </span><br class="">> Sleep Duration Mean Error % Error<br class="">><span class="Apple-converted-space"> </span><br class="">> 1ns 668ns 66831.9<br class="">><span class="Apple-converted-space"> </span><br class="">> 10ns 672ns 6723.7<br class="">><span class="Apple-converted-space"> </span><br class="">> 100ns 557ns 557.6<br class="">><span class="Apple-converted-space"> </span><br class="">> 1µs 57.749µs 5774.9<br class="">><span class="Apple-converted-space"> </span><br class="">> 10µs 63.063µs 630.6<br class="">><span class="Apple-converted-space"> </span><br class="">> 100µs 67.737µs 67.7<br class="">><span class="Apple-converted-space"> </span><br class="">> 1ms 153.978µs 15.4<br class="">><span class="Apple-converted-space"> </span><br class="">> 10ms 169.709µs 1.7<br class="">><span class="Apple-converted-space"> </span><br class="">> 100ms 186.685µs 0.2<br class="">><span class="Apple-converted-space"> </span><br class="">> 200ms 176.859µs 0.1<br class="">><span class="Apple-converted-space"> </span><br class="">> 500ms 177.271µs 0.0<br class="">><span class="Apple-converted-space"> </span><br class="">> ><br class="">> > --<br class="">> > Mikael Abrahamsson email: <a href="mailto:swmike@swm.pp.se" class="">swmike@swm.pp.se</a><br class="">> > _______________________________________________<br class="">> ><br class="">> ><br class="">> > Bloat mailing list<br class="">> > <a href="mailto:Bloat@lists.bufferbloat.net" class="">Bloat@lists.bufferbloat.net</a><br class="">> > <a href="https://lists.bufferbloat.net/listinfo/bloat" class="">https://lists.bufferbloat.net/listinfo/bloat</a><br class="">> ><br class="">> ><br class="">> ><br class="">> > _______________________________________________<br class="">> > Bloat mailing list<br class="">> > <a href="mailto:Bloat@lists.bufferbloat.net" class="">Bloat@lists.bufferbloat.net</a><br class="">> > <a href="https://lists.bufferbloat.net/listinfo/bloat" class="">https://lists.bufferbloat.net/listinfo/bloat</a><br class="">></div></div></font><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class=""></span><span class="BEGIN-ANTISPAM-VOTING-LINKS" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"></span><div class="BEGIN-ANTISPAM-VOTING-LINKS" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; display: inline;"><div class=""><span style="font-size: inherit; font-style: normal; font-weight: normal; background-color: white; display: inline;" class=""><hr class=""><br class=""><a rel="nofollow" target="canit_note" href="https://portal.roaringpenguin.co.uk/canit/b.php?c=s&i=03UJaRTkO&m=5027f7184ff5&rlm=pnsol-com&t=20171212" class="">Spam</a><br class=""><a rel="nofollow" target="canit_note" href="https://portal.roaringpenguin.co.uk/canit/b.php?c=n&i=03UJaRTkO&m=5027f7184ff5&rlm=pnsol-com&t=20171212" class="">Not spam</a><br class=""><a rel="nofollow" target="canit_note" href="https://portal.roaringpenguin.co.uk/canit/b.php?c=f&i=03UJaRTkO&m=5027f7184ff5&rlm=pnsol-com&t=20171212" class="">Forget previous vote</a><br class=""></span></div></div><span class="END-ANTISPAM-VOTING-LINKS" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"></span><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class=""></span><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">_______________________________________________</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Bloat mailing list</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><a href="mailto:Bloat@lists.bufferbloat.net" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">Bloat@lists.bufferbloat.net</a><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><a href="https://lists.bufferbloat.net/listinfo/bloat" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">https://lists.bufferbloat.net/listinfo/bloat</a></div></blockquote></div><br class=""></body></html>