<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">On Fri, Apr 5, 2019 at 3:42 AM Dave Taht <<a href="mailto:dave.taht@gmail.com" target="_blank">dave.taht@gmail.com</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">I see from the iccrg preso at 7 minutes 55 s in, that there is a test<br>
described as:<br>
<br>
20 BBRv2 flows<br>
starting each 100ms, 1G, 1ms<br>
Linux codel with ECN ce_threshold at 242us sojurn time.<br></blockquote><div><br></div><div>Hi, Dave! Thanks for your e-mail.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
I interpret this as<br>
<br>
20 flows, starting 100ms apart<br>
on a 1G link<br>
with a 1ms transit time<br>
and linux codel with ce_threshold 242us<br></blockquote><div><br></div><div>Yes, except the 1ms is end-to-end two-way propagation time.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
0) This is iperf? There is no crypto?<br></blockquote><div><br></div><div>Each flow is a netperf TCP stream, with no crypto.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
1) "sojourn time" not as as setting the codel target to 242us?<br>
<br>
I tend to mentally tie the concept of sojourn time to the target<br>
variable, not ce_threshold<br></blockquote><div><br></div>Right. I didn't mean setting the codel target to 242us. Where the slide says "Linux codel with ECN ce_threshold at 242us sojourn time" I literally mean a Linux machine with a codel qdisc configured as:</div><div class="gmail_quote"><span style="color:rgb(0,0,0);font-family:"Source Sans Pro";font-size:13.3333px"><br></span></div> codel ce_threshold 242us<div class="gmail_quote"><div><br></div><div>This is using the ce_threshold feature added in:</div><div> <a href="https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=80ba92fa1a92dea1" target="_blank">https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=80ba92fa1a92dea1</a><br></div><div><br></div><div>... for which the commit message says:</div><div><br></div><div>"<span style="color:rgb(51,51,51);font-family:monospace;font-size:13.3333px;white-space:pre-wrap">A DCTCP enabled egress port simply have a queue occupancy threshold</span></div><span style="color:rgb(51,51,51);font-family:monospace;font-size:13.3333px;white-space:pre-wrap">above which ECT packets get CE mark.
In codel language this translates to a sojourn time, so that one doesn't
have to worry about bytes or bandwidth but delays."</span></div><div class="gmail_quote"><font color="#333333" face="monospace"><span style="font-size:13.3333px;white-space:pre-wrap"><br></span></font><div>The 242us comes from the seriailization delay for 20 packets at 1Gbps.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
2) In our current SCE work we have repurposed ce_threshold to do sce<br>
instead (to save on cpu and also to make it possible to fiddle without<br>
making a userspace api change). Should we instead create a separate<br>
sce_threshold option to allow for backward compatible usage?<br></blockquote><div><br></div><div>Yes, you would need to maintain the semantics of ce_threshold for backwards compatibility for users who are relying on the current semantics. IMHO your suggestion to use a separate sce_threshold sounds like the way to go, if adding SCE to qdiscs in Linux.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
3) Transit time on your typical 1G link is actually 13us for a big<br>
packet, why 1ms?<br></blockquote><div><br></div><div>The 1ms is the path two-way propagation delay ("min RTT"). We run a range of RTTs in our tests, and the graph happens to be for an RTT of 1ms.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
is that 1ms from netem?<br></blockquote><div><br></div><div>Yes.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
4) What is the topology here?<br>
<br>
host -> qdisc -> wire -> host?<br>
<br>
host -> qdisc -> wire -> router -> host?<br></blockquote><div><br></div><div>Those two won't work with Linux TCP, because putting the qdisc on the sender pulls the qdisc delays inside the TSQ control loop, giving a behavior very different from reality (even CUBIC won't bloat if the network emulation qdiscs are on the sender host). </div><div><br></div><div>What we use for our testing is:</div><div><br></div><div> host -> wire -> qdiscs -> host</div><div><br></div><div>Where "qdiscs" includes netem and whatever AQM is in use, if any.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
5) What was the result with fq_codel instead?<br></blockquote><div><br></div><div>With fq_codel and the same ECN marking threshold (fq_codel ce_threshold 242us), we see slightly smoother fairness properties (not surprising) but with slightly higher latency.</div><div><div><font face="monospace, monospace"><br></font></div>The basic summary:</div><div><br><div><font face="monospace, monospace">retransmits: 0</font></div><div><font face="monospace, monospace">flow throughput: [46.77 .. 51.48]</font></div><div><font face="monospace, monospace">RTT samples at various percentiles:</font></div><div><font face="monospace, monospace"> % | RTT (ms)</font></div><div><font face="monospace, monospace">------+---------</font></div><div><font face="monospace, monospace"> 0 1.009</font></div><div><font face="monospace, monospace"> 50 1.334</font></div><div><font face="monospace, monospace"> 60 1.416</font></div><div><font face="monospace, monospace"> 70 1.493</font></div><div><font face="monospace, monospace"> 80 1.569</font></div><div><font face="monospace, monospace"> 90 1.655</font></div><div><font face="monospace, monospace"> 95 1.725</font></div><div><font face="monospace, monospace"> 99 1.902</font></div><div><font face="monospace, monospace"> 99.9 2.328</font></div><div><font face="monospace, monospace"> 100 6.414</font></div></div><div><font face="monospace, monospace"><br></font></div>Bandwidth share graphs are attached. (Hopefully the graphs will make it through various lists; if not, you can check the <a href="https://groups.google.com/d/msg/bbr-dev/34nWGWLUjp4/nBxQRV01BgAJ">bbr-dev group thread</a>.)<br><br>best,<br>neal</div><div class="gmail_quote"><br></div></div></div></div>