<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Jan 8, 2019, at 11:33 PM, Dave Taht <<a href="mailto:dave.taht@gmail.com" class="">dave.taht@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">On Tue, Jan 8, 2019 at 2:01 PM Pete Heist <</span><a href="mailto:pete@heistp.net" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">pete@heistp.net</a><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">> wrote:</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br class="">I should have done that:<span class="Apple-converted-space"> </span><a href="https://www.heistp.net/downloads/htb_split_gso_patched/" class="">https://www.heistp.net/downloads/htb_split_gso_patched/</a><br class=""><br class="">Note that I changed the names in the plots to match the convention of my first email, but it should be clear which is which and I left all plots in. The text output is there too as I sometimes like to open several up in different browser tabs and switch between tabs to compare values.<br class=""><br class="">It looks like about 100 usec to me. Throughput also looks consistently about 0.3 Mbit higher (~1.3%) in the split results.<br class=""></blockquote><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">My guess is with ecn on would have the highest latency and the same</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">throughput. ?</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""></div></blockquote><div><br class=""></div><div>I see a slight increase in icmp/udp rtt (1.04ms-1.02ms = 20us) and a slight increase in throughput (180.12Mbit - 179.50Mbit = 620kbit).</div><div><br class=""></div><div><a href="https://www.heistp.net/downloads/htb_cakep2_ecnon/" class="">https://www.heistp.net/downloads/htb_cakep2_ecnon/</a></div><div><a href="https://www.heistp.net/downloads/htb_cakep2_ecnon2/" class="">https://www.heistp.net/downloads/htb_cakep2_ecnon2/</a></div><div><br class=""></div><div>These results are not five sigma. :)</div><div><br class=""></div><blockquote type="cite" class=""><div class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">Since we started this effort in an era when seconds of added latency</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">was common, a mere 100us improvement seems insignificant, except that</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">that's a 10% improvement over the present-day baseline, and *that's</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">worth it*. ;) This is also a function of the number of flows, kernel</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""></div></blockquote><div><br class=""></div><div>Who knows how many timeouts are avoided in the future, just because of this 100us. :)</div><br class=""><blockquote type="cite" class=""><div class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">I have noticed that BQL's values can get quite large with cake doing</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">the shaping, btw, much larger than they do with htb.</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""></div></blockquote></div><br class=""><div class="">Wonder why that is.</div><div class=""><br class=""></div><div class="">So far it’s harder to use cake’s shaper in the current setup I’m working on, because it appears “better" to have eth0 and eth0.3300 under one link sharing hierarchy for eth0:</div><div class=""><br class=""></div><div class="">tc qdisc add dev eth0 root handle 1: htb default 1<br class="">tc class add dev eth0 parent 1: classid 1:1 htb rate $RATE<br class="">tc class add dev eth0 parent 1: classid 1:2 htb rate $RATE<br class="">tc qdisc add dev eth0 parent 1:1 cake besteffort<br class="">tc qdisc add dev eth0 parent 1:2 cake besteffort<br class="">tc filter add dev eth0 parent 1:0 prio 1 protocol all \</div><div class=""><span class="Apple-tab-span" style="white-space:pre"> </span>basic match not "meta(vlan mask 0xfff gt 0x0)" flowid 1:1<br class="">tc filter add dev eth0 parent 1:0 prio 2 protocol all \</div><div class=""><span class="Apple-tab-span" style="white-space:pre"> </span>basic match "meta(vlan mask 0xfff eq `printf \"0x%x\" $VLAN_TAG`)" flowid 1:2<br class=""><br class=""></div><div class="">but I can’t do that with cake’s shaper. It’s better in the sense that when you’re out of CPU, latency doesn’t increase suddenly. I _can_ use cake’s shaper by adding cake to the VLAN interface and filtering out vlan traffic from the main interface:</div><div class=""><br class=""></div><div class="">tc qdisc add dev eth0 root handle 1: prio bands 2 priomap 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1<br class="">tc qdisc add dev eth0 parent 1:1 handle 10: cake besteffort bandwidth $RATE<br class="">tc filter add dev eth0 parent 1:0 prio 1 protocol all \<br class=""><span class="Apple-tab-span" style="white-space:pre"> </span>basic match not "meta(vlan mask 0xfff gt 0x0)" flowid 10:1<br class="">tc qdisc add dev eth0.3300 root cake besteffort bandwidth $RATE<br class=""><br class=""></div><div class="">but when you’re out of CPU, you get starvation and inter-flow latency increases. So so far, hfsc or htb is working better in this case. It might be something to think about for an “ISP Cake”…</div><div class=""><br class=""></div></body></html>