<div dir="ltr"><div dir="auto">Sure thing. Defining some multivariate signals using "non-bulk flow" or "realistic" traffic scenarios that could be automated (and a proxy for user QoE) would be very useful for L2 driver, MAC/PHY, and AP scheduling engineers allowing them to provide the best quality packet forwarding plane products possible for the transport layers and those engineering teams.<br><br>Bob </div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Aug 29, 2022, 1:07 PM Neal Cardwell <<a href="mailto:ncardwell@google.com" target="_blank">ncardwell@google.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Thanks for the pointers, Bob.</div><div><br></div><div>best regards,</div><div>neal</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Aug 29, 2022 at 12:47 PM 'Bob McMahon' via BBR Development <<a href="mailto:bbr-dev@googlegroups.com" rel="noreferrer" target="_blank">bbr-dev@googlegroups.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Thanks Neal. You might want to check out the <a href="https://sourceforge.net/p/iperf2/code/ci/master/tree/flows/" rel="noreferrer" target="_blank">flows released as iperf 2.</a> Basically instantiate flows and run them. There typically is a controller running python3 (v 3.10 or better) that uses ssh pipes to DUTs. The design is <a href="https://en.wikipedia.org/wiki/Event-driven_programming" rel="noreferrer" target="_blank">event driven</a> and utilizes python's <a href="https://docs.python.org/3/library/asyncio.html" rel="noreferrer" target="_blank">asyncio</a> which is quite powerful. The DUTs just need iperf2 and ssh.<br><br>The code is at an alpha level and we're looking for broader industry support and contributions. Both in realtime plotting but also in things like multivariate regression detection using statistical process controls (SPC) e.g. <a href="https://www.itl.nist.gov/div898/handbook/pmc/section3/pmc341.htm" rel="noreferrer" target="_blank">Hoteling</a>. There is some crude clustering code around latency too which uses <a href="https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test" rel="noreferrer" target="_blank">Kolmogorov-Smirnov </a>distance matrices per the histograms.<br><br>A suggestion is that those in developer and control test roles synchronize their device clocks with PTP. Iperf 2 supports one way delay (OWD) calculations but these only work if the clocks are sync'd. These in turn can be used per <a href="https://en.wikipedia.org/wiki/Little%27s_law" rel="noreferrer" target="_blank">Little's law</a> to calculate effective average queue depth, though this typically assumes a steady state measurement.<br><br>Bob </div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Aug 28, 2022 at 4:54 PM Neal Cardwell <<a href="mailto:ncardwell@google.com" rel="noreferrer" target="_blank">ncardwell@google.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">If you are talking about the screenshot of the UI at <a href="https://github.com/google/transperf" rel="noreferrer noreferrer" target="_blank">https://github.com/google/transperf</a>, yes, that particular test is a simple bulk flow test to show a simple case to give a sense of what the UI looks like. :-)<div><br></div><div>We use a few different approaches that can examine dynamic flows causing packet loss: <br><div><br></div><div>(1) The test configuration language is Python, so you can construct arbitrarily fancy dynamic flow scenarios with arbitrary numbers of flows starting and stopping at arbitrary times.</div><div><br></div><div>(2) The tests can also use netperf command line options to run periodic short transfers. (And we welcome patches to integrate support for other tools.)</div><div><br></div><div>(3) We also run a fair number of tests for robustness to loss just using randomly injected packet loss (using netem).</div><div><br></div><div>These are just some of the approaches we have used, and I don't claim that these are the only or best approaches to look at this. :-)</div><div><br></div><div>cheers,</div><div>neal</div><div><br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Aug 28, 2022 at 6:39 PM Bob McMahon <<a href="mailto:bob.mcmahon@broadcom.com" rel="noreferrer" target="_blank">bob.mcmahon@broadcom.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Neal,<div><br></div><div>These look like steady-state bulk flow tests unless I'm missing something.<br><br>Bob</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Aug 28, 2022 at 11:43 AM Neal Cardwell <<a href="mailto:ncardwell@google.com" rel="noreferrer" target="_blank">ncardwell@google.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto"><div dir="auto">Sure. For testing these kinds of properties of the BBR algorithm we use various transperf test cases. The transperf tool is something Soheil Hassas Yeganeh and our team cooked up and open-sourced here:</div><div dir="auto"><div dir="auto"><br></div><div dir="auto"> <a href="https://github.com/google/transperf" rel="noreferrer noreferrer" target="_blank">https://github.com/google/transperf</a></div><div dir="auto"><br></div><div dir="auto">Best regards,</div><div dir="auto">neal</div></div><div dir="auto"><br><div class="gmail_quote" dir="auto"><div dir="ltr" class="gmail_attr">On Sat, Aug 27, 2022, 4:43 PM Bob McMahon <<a href="mailto:bob.mcmahon@broadcom.com" rel="noreferrer noreferrer" target="_blank">bob.mcmahon@broadcom.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Curious to what you're doing during development, if you can share?<br><br>Thanks,<br>Bob</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Aug 27, 2022 at 7:44 AM Neal Cardwell <<a href="mailto:ncardwell@google.com" rel="noreferrer noreferrer noreferrer" target="_blank">ncardwell@google.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Bob,<div><br></div><div>Good question. I can imagine a number of different techniques to generate and measure the traffic flows for this kind of study, and don't have any particular suggestions.</div><div><br></div><div>neal</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Aug 26, 2022 at 4:54 PM Bob McMahon <<a href="mailto:bob.mcmahon@broadcom.com" rel="noreferrer noreferrer noreferrer" target="_blank">bob.mcmahon@broadcom.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Neal,<br><br>Any thoughts on tooling to generate and measure the traffic flows BBR is designed to optimize? I've been adding some low duty cycle support in <a href="https://sourceforge.net/projects/iperf2/" rel="noreferrer noreferrer noreferrer" target="_blank">iperf 2</a> with things like <a href="https://iperf2.sourceforge.io/iperf-manpage.html" rel="noreferrer noreferrer noreferrer" target="_blank">--bounceback and --burst-period and --burst-period</a>. We could pull the size and period from a known distribution or distributions though not sure what to pick.<br><br>Thanks,<br>Bob<br><br>Bob</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Aug 26, 2022 at 6:36 AM 'Neal Cardwell' via BBR Development <<a href="mailto:bbr-dev@googlegroups.com" rel="noreferrer noreferrer noreferrer" target="_blank">bbr-dev@googlegroups.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Yes, I agree the assumptions are key here. One key aspect of this paper is that it focuses on the steady-state behavior of bulk flows.<div><br></div><div>Once you allow for short flows (like web pages, RPCs, etc) to dynamically enter and leave a bottleneck, the considerations become different. As is well-known, Reno/CUBIC will starve themselves if new flows enter and cause loss too frequently. For CUBIC, for a somewhat typical 30ms broadband path with a flow fair share of 25 Mbit/sec, if new flows enter and cause loss more frequently than roughly every 2 seconds then CUBIC will not be able to utilize its fair share. For a high-speed WAN path, with 100ms RTT and fair share of 10 Gbit/sec, if new flows enter and cause loss more frequently than roughly every 40 seconds then CUBIC will not be able to utilize its fair share. Basically, loss-based CC can starve itself in some very typical kinds of dynamic scenarios that happen in the real world.</div><div><br></div><div>BBR is not trying to maintain a higher throughput than CUBIC in these kinds of scenarios with steady-state bulk flows. BBR is trying to be robust to the kinds of random packet loss that happen in the real world when there are flows dynamically entering/leaving a bottleneck.</div><div><br></div><div>cheers,</div><div>neal</div><div><br></div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Aug 25, 2022 at 8:01 PM Dave Taht via Bloat <<a href="mailto:bloat@lists.bufferbloat.net" rel="noreferrer noreferrer noreferrer" target="_blank">bloat@lists.bufferbloat.net</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">I rather enjoyed this one. I can't help but wonder what would happen<br>
if we plugged some different assumptions into their model.<br>
<br>
<a href="https://www.comp.nus.edu.sg/~bleong/publications/imc2022-nash.pdf" rel="noreferrer noreferrer noreferrer noreferrer" target="_blank">https://www.comp.nus.edu.sg/~bleong/publications/imc2022-nash.pdf</a><br>
<br>
-- <br>
FQ World Domination pending: <a href="https://blog.cerowrt.org/post/state_of_fq_codel/" rel="noreferrer noreferrer noreferrer noreferrer" target="_blank">https://blog.cerowrt.org/post/state_of_fq_codel/</a><br>
Dave Täht CEO, TekLibre, LLC<br>
_______________________________________________<br>
Bloat mailing list<br>
<a href="mailto:Bloat@lists.bufferbloat.net" rel="noreferrer noreferrer noreferrer" target="_blank">Bloat@lists.bufferbloat.net</a><br>
<a href="https://lists.bufferbloat.net/listinfo/bloat" rel="noreferrer noreferrer noreferrer noreferrer" target="_blank">https://lists.bufferbloat.net/listinfo/bloat</a><br>
</blockquote></div>
<p></p>
-- <br>
You received this message because you are subscribed to the Google Groups "BBR Development" group.<br>
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="mailto:bbr-dev+unsubscribe@googlegroups.com" rel="noreferrer noreferrer noreferrer" target="_blank">bbr-dev+unsubscribe@googlegroups.com</a>.<br>
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/bbr-dev/CADVnQykKbnxpNcpuZATug_4VLhV1%3DaoTTQE2263o8HF9ye_TQg%40mail.gmail.com?utm_medium=email&utm_source=footer" rel="noreferrer noreferrer noreferrer" target="_blank">https://groups.google.com/d/msgid/bbr-dev/CADVnQykKbnxpNcpuZATug_4VLhV1%3DaoTTQE2263o8HF9ye_TQg%40mail.gmail.com</a>.<br>
</blockquote></div>
<br>
<span style="background-color:rgb(255,255,255)"><font size="2">This electronic communication and the information and any files transmitted with it, or attached to it, are confidential and are intended solely for the use of the individual or entity to whom it is addressed and may contain information that is confidential, legally privileged, protected by privacy laws, or otherwise restricted from disclosure to anyone else. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended recipient, you are hereby notified that any use, copying, distributing, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited. If you received this e-mail in error, please return the e-mail to the sender, delete it from your computer, and destroy any printed copy of it.</font></span></blockquote></div>
</blockquote></div>
<br>
<span style="background-color:rgb(255,255,255)"><font size="2">This electronic communication and the information and any files transmitted with it, or attached to it, are confidential and are intended solely for the use of the individual or entity to whom it is addressed and may contain information that is confidential, legally privileged, protected by privacy laws, or otherwise restricted from disclosure to anyone else. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended recipient, you are hereby notified that any use, copying, distributing, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited. If you received this e-mail in error, please return the e-mail to the sender, delete it from your computer, and destroy any printed copy of it.</font></span></blockquote></div></div></div>
</blockquote></div>
<br>
<span style="background-color:rgb(255,255,255)"><font size="2">This electronic communication and the information and any files transmitted with it, or attached to it, are confidential and are intended solely for the use of the individual or entity to whom it is addressed and may contain information that is confidential, legally privileged, protected by privacy laws, or otherwise restricted from disclosure to anyone else. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended recipient, you are hereby notified that any use, copying, distributing, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited. If you received this e-mail in error, please return the e-mail to the sender, delete it from your computer, and destroy any printed copy of it.</font></span></blockquote></div>
</blockquote></div>
<br>
<span style="background-color:rgb(255,255,255)"><font size="2">This electronic communication and the information and any files transmitted with it, or attached to it, are confidential and are intended solely for the use of the individual or entity to whom it is addressed and may contain information that is confidential, legally privileged, protected by privacy laws, or otherwise restricted from disclosure to anyone else. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended recipient, you are hereby notified that any use, copying, distributing, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited. If you received this e-mail in error, please return the e-mail to the sender, delete it from your computer, and destroy any printed copy of it.</font></span>
<p></p>
-- <br>
You received this message because you are subscribed to the Google Groups "BBR Development" group.<br>
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="mailto:bbr-dev+unsubscribe@googlegroups.com" rel="noreferrer" target="_blank">bbr-dev+unsubscribe@googlegroups.com</a>.<br>
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/bbr-dev/CAHb6Lvqi9%3Dr0H6uNekoCs8tm450Bd-6%2BA9v7XMszX73h7yVMPg%40mail.gmail.com?utm_medium=email&utm_source=footer" rel="noreferrer" target="_blank">https://groups.google.com/d/msgid/bbr-dev/CAHb6Lvqi9%3Dr0H6uNekoCs8tm450Bd-6%2BA9v7XMszX73h7yVMPg%40mail.gmail.com</a>.<br>
</blockquote></div>
</blockquote></div>
<br>
<span style="background-color:rgb(255,255,255)"><font size="2">This electronic communication and the information and any files transmitted with it, or attached to it, are confidential and are intended solely for the use of the individual or entity to whom it is addressed and may contain information that is confidential, legally privileged, protected by privacy laws, or otherwise restricted from disclosure to anyone else. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended recipient, you are hereby notified that any use, copying, distributing, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited. If you received this e-mail in error, please return the e-mail to the sender, delete it from your computer, and destroy any printed copy of it.</font></span>