<div dir="ltr">"Control at endpoints benefits greatly from even small amounts of information supplied by the network about the degree of congestion present on the path."<br><br>Agreed. The ECN mechanism seems like a shared thermostat in a building. It's basically an on/off where everyone is trying to set the temperature. It does affect, in a non-linear manner, but still an effect. Better than a thermostat set at infinity or 0 Kelvin for sure.<br><br>I find the assumption that congestion occurs "in network" as not always true. Taking OWD measurements with read side rate limiting suggests that equally important to mitigating bufferbloat driven latency using congestion signals is to make sure apps read "fast enough" whatever that means. I rarely hear about how important it is for apps to prioritize reads over open sockets. Not sure why that's overlooked and bufferbloat gets all the attention. I'm probably missing something.<br><br>Bob</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Jul 13, 2021 at 12:15 AM Amr Rizk <<a href="mailto:amr@rizk.com.de">amr@rizk.com.de</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Ben, <br>
<br>
it depends on what one tries to measure. Doing a rate scan using UDP (to measure latency distributions under load) is the best thing that we have but without actually knowing how resources are shared (fair share as in WiFi, FIFO as nearly everywhere else) it becomes very difficult to interpret the results or provide a proper argument on latency. You are right - TCP stats are a proxy for user experience but I believe they are difficult to reproduce (we are always talking about very short TCP flows - the infinite TCP flow that converges to a steady behavior is purely academic).<br>
<br>
By the way, Little's law is a strong tool when it comes to averages. To be able to say more (e.g. 1% of the delays is larger than x) one requires more information (e.g. the traffic - On-OFF pattern) see [1]. I am not sure when does such information readily exist. <br>
<br>
Best<br>
Amr <br>
<br>
[1] <a href="https://dl.acm.org/doi/10.1145/3341617.3326146" rel="noreferrer" target="_blank">https://dl.acm.org/doi/10.1145/3341617.3326146</a> or if behind a paywall <a href="https://www.dcs.warwick.ac.uk/~florin/lib/sigmet19b.pdf" rel="noreferrer" target="_blank">https://www.dcs.warwick.ac.uk/~florin/lib/sigmet19b.pdf</a><br>
<br>
--------------------------------<br>
Amr Rizk (<a href="mailto:amr.rizk@uni-due.de" target="_blank">amr.rizk@uni-due.de</a>)<br>
University of Duisburg-Essen<br>
<br>
-----Ursprüngliche Nachricht-----<br>
Von: Bloat <<a href="mailto:bloat-bounces@lists.bufferbloat.net" target="_blank">bloat-bounces@lists.bufferbloat.net</a>> Im Auftrag von Ben Greear<br>
Gesendet: Montag, 12. Juli 2021 22:32<br>
An: Bob McMahon <<a href="mailto:bob.mcmahon@broadcom.com" target="_blank">bob.mcmahon@broadcom.com</a>><br>
Cc: <a href="mailto:starlink@lists.bufferbloat.net" target="_blank">starlink@lists.bufferbloat.net</a>; Make-Wifi-fast <<a href="mailto:make-wifi-fast@lists.bufferbloat.net" target="_blank">make-wifi-fast@lists.bufferbloat.net</a>>; Leonard Kleinrock <<a href="mailto:lk@cs.ucla.edu" target="_blank">lk@cs.ucla.edu</a>>; David P. Reed <<a href="mailto:dpreed@deepplum.com" target="_blank">dpreed@deepplum.com</a>>; Cake List <<a href="mailto:cake@lists.bufferbloat.net" target="_blank">cake@lists.bufferbloat.net</a>>; <a href="mailto:codel@lists.bufferbloat.net" target="_blank">codel@lists.bufferbloat.net</a>; cerowrt-devel <<a href="mailto:cerowrt-devel@lists.bufferbloat.net" target="_blank">cerowrt-devel@lists.bufferbloat.net</a>>; bloat <<a href="mailto:bloat@lists.bufferbloat.net" target="_blank">bloat@lists.bufferbloat.net</a>><br>
Betreff: Re: [Bloat] Little's Law mea culpa, but not invalidating my main point<br>
<br>
UDP is better for getting actual packet latency, for sure. TCP is typical-user-experience-latency though, so it is also useful.<br>
<br>
I'm interested in the test and visualization side of this. If there were a way to give engineers a good real-time look at a complex real-world network, then they have something to go on while trying to tune various knobs in their network to improve it.<br>
<br>
I'll let others try to figure out how build and tune the knobs, but the data acquisition and visualization is something we might try to accomplish. I have a feeling I'm not the first person to think of this, however....probably someone already has done such a thing.<br>
<br>
Thanks,<br>
Ben<br>
<br>
On 7/12/21 1:04 PM, Bob McMahon wrote:<br>
> I believe end host's TCP stats are insufficient as seen per the <br>
> "failed" congested control mechanisms over the last decades. I think <br>
> Jaffe pointed this out in<br>
> 1979 though he was using what's been deemed on this thread as "spherical cow queueing theory."<br>
> <br>
> "Flow control in store-and-forward computer networks is appropriate <br>
> for decentralized execution. A formal description of a class of <br>
> "decentralized flow control algorithms" is given. The feasibility of <br>
> maximizing power with such algorithms is investigated. On the <br>
> assumption that communication links behave like M/M/1 servers it is shown that no "decentralized flow control algorithm" can maximize network power. Power has been suggested in the literature as a network performance objective. It is also shown that no objective based only on the users' throughputs and average delay is decentralizable. Finally, a restricted class of algorithms cannot even approximate power."<br>
> <br>
> <a href="https://ieeexplore.ieee.org/document/1095152" rel="noreferrer" target="_blank">https://ieeexplore.ieee.org/document/1095152</a><br>
> <br>
> Did Jaffe make a mistake?<br>
> <br>
> Also, it's been observed that latency is non-parametric in it's <br>
> distributions and computing gaussians per the central limit theorem <br>
> for OWD feedback loops aren't effective. How does one design a control loop around things that are non-parametric? It also begs the question, what are the feed forward knobs that can actually help?<br>
> <br>
> Bob<br>
> <br>
> On Mon, Jul 12, 2021 at 12:07 PM Ben Greear <<a href="mailto:greearb@candelatech.com" target="_blank">greearb@candelatech.com</a> <mailto:<a href="mailto:greearb@candelatech.com" target="_blank">greearb@candelatech.com</a>>> wrote:<br>
> <br>
> Measuring one or a few links provides a bit of data, but seems like if someone is trying to understand<br>
> a large and real network, then the OWD between point A and B needs to just be input into something much<br>
> more grand. Assuming real-time OWD data exists between 100 to 1000 endpoint pairs, has anyone found a way<br>
> to visualize this in a useful manner?<br>
> <br>
> Also, considering something better than ntp may not really scale to 1000+ endpoints, maybe round-trip<br>
> time is only viable way to get this type of data. In that case, maybe clever logic could use things<br>
> like trace-route to get some idea of how long it takes to get 'onto' the internet proper, and so estimate<br>
> the last-mile latency. My assumption is that the last-mile latency is where most of the pervasive<br>
> assymetric network latencies would exist (or just ping 8.8.8.8 which is 20ms from everywhere due to<br>
> $magic).<br>
> <br>
> Endpoints could also triangulate a bit if needed, using some anchor points in the network<br>
> under test.<br>
> <br>
> Thanks,<br>
> Ben<br>
> <br>
> On 7/12/21 11:21 AM, Bob McMahon wrote:<br>
> > iperf 2 supports OWD and gives full histograms for TCP write to read, TCP connect times, latency of packets (with UDP), latency of "frames" with<br>
> > simulated video traffic (TCP and UDP), xfer times of bursts with low duty cycle traffic, and TCP RTT (sampling based.) It also has support for sampling (per<br>
> > interval reports) down to 100 usecs if configured with --enable-fastsampling, otherwise the fastest sampling is 5 ms. We've released all this as open source.<br>
> ><br>
> > OWD only works if the end realtime clocks are synchronized using a "machine level" protocol such as IEEE 1588 or PTP. Sadly, *most data centers don't<br>
> provide<br>
> > sufficient level of clock accuracy and the GPS pulse per second * to colo and vm customers.<br>
> ><br>
> > <a href="https://iperf2.sourceforge.io/iperf-manpage.html" rel="noreferrer" target="_blank">https://iperf2.sourceforge.io/iperf-manpage.html</a><br>
> ><br>
> > Bob<br>
> ><br>
> > On Mon, Jul 12, 2021 at 10:40 AM David P. Reed <<a href="mailto:dpreed@deepplum.com" target="_blank">dpreed@deepplum.com</a> <mailto:<a href="mailto:dpreed@deepplum.com" target="_blank">dpreed@deepplum.com</a>> <mailto:<a href="mailto:dpreed@deepplum.com" target="_blank">dpreed@deepplum.com</a><br>
> <mailto:<a href="mailto:dpreed@deepplum.com" target="_blank">dpreed@deepplum.com</a>>>> wrote:<br>
> ><br>
> ><br>
> > On Monday, July 12, 2021 9:46am, "Livingood, Jason" <<a href="mailto:Jason_Livingood@comcast.com" target="_blank">Jason_Livingood@comcast.com</a> <mailto:<a href="mailto:Jason_Livingood@comcast.com" target="_blank">Jason_Livingood@comcast.com</a>><br>
> <mailto:<a href="mailto:Jason_Livingood@comcast.com" target="_blank">Jason_Livingood@comcast.com</a> <mailto:<a href="mailto:Jason_Livingood@comcast.com" target="_blank">Jason_Livingood@comcast.com</a>>>> said:<br>
> ><br>
> > > I think latency/delay is becoming seen to be as important certainly, if not a more direct proxy for end user QoE. This is all still evolving and I<br>
> have<br>
> > to say is a super interesting & fun thing to work on. :-)<br>
> ><br>
> > If I could manage to sell one idea to the management hierarchy of communications industry CEOs (operators, vendors, ...) it is this one:<br>
> ><br>
> > "It's the end-to-end latency, stupid!"<br>
> ><br>
> > And I mean, by end-to-end, latency to complete a task at a relevant layer of abstraction.<br>
> ><br>
> > At the link level, it's packet send to packet receive completion.<br>
> ><br>
> > But at the transport level including retransmission buffers, it's datagram (or message) origination until the acknowledgement arrives for that<br>
> message being<br>
> > delivered after whatever number of retransmissions, freeing the retransmission buffer.<br>
> ><br>
> > At the WWW level, it's mouse click to display update corresponding to completion of the request.<br>
> ><br>
> > What should be noted is that lower level latencies don't directly predict the magnitude of higher-level latencies. But longer lower level latencies<br>
> almost<br>
> > always amplfify higher level latencies. Often non-linearly.<br>
> ><br>
> > Throughput is very, very weakly related to these latencies, in contrast.<br>
> ><br>
> > The amplification process has to do with the presence of queueing. Queueing is ALWAYS bad for latency, and throughput only helps if it is in exactly the<br>
> > right place (the so-called input queue of the bottleneck process, which is often a link, but not always).<br>
> ><br>
> > Can we get that slogan into Harvard Business Review? Can we get it taught in Managerial Accounting at HBS? (which does address logistics/supply chain<br>
> queueing).<br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> ><br>
> > This electronic communication and the information and any files transmitted with it, or attached to it, are confidential and are intended solely for the<br>
> use of<br>
> > the individual or entity to whom it is addressed and may contain information that is confidential, legally privileged, protected by privacy laws, or<br>
> otherwise<br>
> > restricted from disclosure to anyone else. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended<br>
> recipient,<br>
> > you are hereby notified that any use, copying, distributing, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited. If you<br>
> > received this e-mail in error, please return the e-mail to the sender, delete it from your computer, and destroy any printed copy of it.<br>
> <br>
> <br>
> -- <br>
> Ben Greear <<a href="mailto:greearb@candelatech.com" target="_blank">greearb@candelatech.com</a> <mailto:<a href="mailto:greearb@candelatech.com" target="_blank">greearb@candelatech.com</a>>><br>
> Candela Technologies Inc <a href="http://www.candelatech.com" rel="noreferrer" target="_blank">http://www.candelatech.com</a><br>
> <br>
> <br>
> This electronic communication and the information and any files <br>
> transmitted with it, or attached to it, are confidential and are <br>
> intended solely for the use of the individual or entity to whom it is <br>
> addressed and may contain information that is confidential, legally <br>
> privileged, protected by privacy laws, or otherwise restricted from disclosure to anyone else. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended recipient, you are hereby notified that any use, copying, distributing, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited. If you received this e-mail in error, please return the e-mail to the sender, delete it from your computer, and destroy any printed copy of it.<br>
<br>
<br>
--<br>
Ben Greear <<a href="mailto:greearb@candelatech.com" target="_blank">greearb@candelatech.com</a>><br>
Candela Technologies Inc <a href="http://www.candelatech.com" rel="noreferrer" target="_blank">http://www.candelatech.com</a><br>
<br>
_______________________________________________<br>
Bloat mailing list<br>
<a href="mailto:Bloat@lists.bufferbloat.net" target="_blank">Bloat@lists.bufferbloat.net</a><br>
<a href="https://lists.bufferbloat.net/listinfo/bloat" rel="noreferrer" target="_blank">https://lists.bufferbloat.net/listinfo/bloat</a><br>
<br>
</blockquote></div>
<br>
<span style="background-color:rgb(255,255,255)"><font size="2">This electronic communication and the information and any files transmitted with it, or attached to it, are confidential and are intended solely for the use of the individual or entity to whom it is addressed and may contain information that is confidential, legally privileged, protected by privacy laws, or otherwise restricted from disclosure to anyone else. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended recipient, you are hereby notified that any use, copying, distributing, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited. If you received this e-mail in error, please return the e-mail to the sender, delete it from your computer, and destroy any printed copy of it.</font></span>