<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">I’m also finding out how simple it is to miss one little thing when looking at hundreds of test runs. Finding the “god metric” for rrul would make life easier...</div><br class=""><div><blockquote type="cite" class=""><div class="">On Nov 27, 2017, at 6:38 PM, Dave Taht <<a href="mailto:dave.taht@gmail.com" class="">dave.taht@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">On Mon, Nov 27, 2017 at 9:34 AM, Sebastian Moeller <</span><a href="mailto:moeller0@gmx.de" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">moeller0@gmx.de</a><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">> wrote:</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">But 444.35 + 443.65 = 888, no?<br class=""></blockquote><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">My bad. I miss-read the test setup. Pre-coffee here, though, that</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">caused an adrenalin spike.</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Yea! per host fairness 1v12! and correct bandwidth on this cpu. :whew:</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br class=""><blockquote type="cite" class="">On Nov 27, 2017, at 18:33, Dave Taht <<a href="mailto:dave.taht@gmail.com" class="">dave.taht@gmail.com</a>> wrote:<br class=""><br class="">georgios<br class=""><br class="">the result you got was "fair", but you shoul have seen something<br class="">closer to 900mbit than 400.<br class=""><br class="">On Mon, Nov 27, 2017 at 8:17 AM, Georgios Amanakis <<a href="mailto:gamanakis@gmail.com" class="">gamanakis@gmail.com</a>> wrote:<br class=""><blockquote type="cite" class="">Dear Pete,<br class=""><br class="">I am trying to replicate the unfair behaviour you are seeing with<br class="">dual-{src,dst}host, albeit on different hardware and I am getting a fair<br class="">distribution. Hardware are Xeon E3-1220Lv2 (router), i3-3110M(Clients). All<br class="">running Archlinux, latest cake and patched iproute2-4.14.1, connected with<br class="">Gbit ethernet, TSO/GSO/GRO enabled.<br class=""><br class="">Qdisc setup:<br class="">----------------<br class="">Router:<br class="">qdisc cake 8003: dev ens4 root refcnt 2 bandwidth 900Mbit diffserv3<br class="">dual-dsthost rtt 100.0ms raw<br class=""><br class="">Client A(kernel default):<br class="">qdisc fq_codel 0: dev eno2 root refcnt 2 limit 10240p flows 1024 quantum<br class="">1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn<br class=""><br class="">Client B (kernel default):<br class="">qdisc fq_codel 0: dev enp1s0 root refcnt 2 limit 10240p flows 1024 quantum<br class="">1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn<br class="">----------------<br class=""><br class=""><br class="">Cli:<br class="">----------------<br class="">Router:<br class="">netserver &<br class=""><br class="">Client A:<br class="">flent tcp_1down -H router<br class=""><br class="">Client B:<br class="">flent tcp_12down -H router<br class="">----------------<br class=""><br class=""><br class="">Results:<br class="">----------------<br class="">Router:<br class="">qdisc cake 8003: root refcnt 2 bandwidth 900Mbit diffserv3 dual-dsthost rtt<br class="">100.0ms raw<br class="">Sent 7126680117 bytes 4725904 pkt (dropped 10, overlimits 4439745 requeues<br class="">0)<br class="">backlog 0b 0p requeues 0<br class="">memory used: 1224872b of 15140Kb<br class="">capacity estimate: 900Mbit<br class=""> Bulk Best Effort Voice<br class="">thresh 56250Kbit 900Mbit 225Mbit<br class="">target 5.0ms 5.0ms 5.0ms<br class="">interval 100.0ms 100.0ms 100.0ms<br class="">pk_delay 14us 751us 7us<br class="">av_delay 2us 642us 1us<br class="">sp_delay 1us 1us 1us<br class="">pkts 109948 4601651 14315<br class="">bytes 160183242 6964893773 1618242<br class="">way_inds 0 21009 0<br class="">way_miss 160 188 5<br class="">way_cols 0 0 0<br class="">drops 0 10 0<br class="">marks 0 0 0<br class="">ack_drop 0 0 0<br class="">sp_flows 0 1 1<br class="">bk_flows 1 0 0<br class="">un_flows 0 0 0<br class="">max_len 7570 68130 1022<br class=""><br class=""><br class="">Client A:<br class=""> avg median # data pts<br class="">Ping (ms) ICMP : 0.11 0.08 ms 350<br class="">TCP download : 443.65 430.38 Mbits/s 301<br class=""><br class=""><br class="">Client B:<br class=""> avg median # data pts<br class="">Ping (ms) ICMP : 0.09 0.06 ms 350<br class="">TCP download avg : 37.03 35.87 Mbits/s 301<br class="">TCP download sum : 444.35 430.40 Mbits/s 301<br class="">TCP download::1 : 37.00 35.87 Mbits/s 301<br class="">TCP download::10 : 37.01 35.87 Mbits/s 301<br class="">TCP download::11 : 37.02 35.87 Mbits/s 301<br class="">TCP download::12 : 37.00 35.87 Mbits/s 301<br class="">TCP download::2 : 37.03 35.87 Mbits/s 301<br class="">TCP download::3 : 36.99 35.87 Mbits/s 301<br class="">TCP download::4 : 37.03 35.87 Mbits/s 301<br class="">TCP download::5 : 37.07 35.87 Mbits/s 301<br class="">TCP download::6 : 37.00 35.87 Mbits/s 301<br class="">TCP download::7 : 37.12 35.87 Mbits/s 301<br class="">TCP download::8 : 37.05 35.87 Mbits/s 301<br class="">TCP download::9 : 37.03 35.87 Mbits/s 301<br class="">----------------<br class=""><br class="">Does this suggest that it is indeed a problem of an underpowered CPU in your<br class="">case?<br class=""><br class="">George<br class=""><br class=""><br class="">On Mon, Nov 27, 2017 at 10:53 AM, Pete Heist <<a href="mailto:peteheist@gmail.com" class="">peteheist@gmail.com</a>> wrote:<br class=""><blockquote type="cite" class=""><br class=""><br class="">On Nov 27, 2017, at 3:48 PM, Jonathan Morton <<a href="mailto:chromatix99@gmail.com" class="">chromatix99@gmail.com</a>><br class="">wrote:<br class=""><br class="">It's not at all obvious how we'd detect that. Packets are staying in the<br class="">queue for less time than the codel target, which is exactly what you'd get<br class="">if you weren't saturated at all.<br class=""><br class="">That makes complete sense when you put it that way. Cake has no way of<br class="">knowing why the input rate is lower than expected, even if it’s part of the<br class="">cause.<br class=""><br class="">I don’t think flent can know this either. It can’t easily know the cause<br class="">for its total output to be lower than expected.<br class=""><br class="">All I know is, this is a common problem in deployments, particularly on<br class="">low-end hardware like ER-Xs, that can be tricky for users to figure out.<br class=""><br class="">I don’t even think monitoring CPU in general would work. The CPU could be<br class="">high because it’s doing other calculations, but there’s still enough for<br class="">cake at a low rate, and there’s no need to warn in that case. I’d be<br class="">interested in any ideas on how to know this is happening in the system as a<br class="">whole. So far, there are just various clues that one needs to piece together<br class="">(no or few drops or marks, less total throughput that expected, high cpu<br class="">without other external usage, etc). Then it needs to be proven with a test.<br class=""><br class="">Anyway thanks, your clue was what I needed! I need to remember to review<br class="">the qdisc stats when something unexpected happens.<br class=""><br class="">_______________________________________________<br class="">Cake mailing list<br class=""><a href="mailto:Cake@lists.bufferbloat.net" class="">Cake@lists.bufferbloat.net</a><br class="">https://lists.bufferbloat.net/listinfo/cake<br class=""><br class=""></blockquote><br class=""><br class="">_______________________________________________<br class="">Cake mailing list<br class=""><a href="mailto:Cake@lists.bufferbloat.net" class="">Cake@lists.bufferbloat.net</a><br class="">https://lists.bufferbloat.net/listinfo/cake<br class=""><br class=""></blockquote><br class=""><br class=""><br class="">--<br class=""><br class="">Dave Täht<br class="">CEO, TekLibre, LLC<br class=""><a href="http://www.teklibre.com" class="">http://www.teklibre.com</a><br class="">Tel: 1-669-226-2619<br class="">_______________________________________________<br class="">Cake mailing list<br class="">Cake@lists.bufferbloat.net<br class="">https://lists.bufferbloat.net/listinfo/cake<br class=""></blockquote><br class=""></blockquote><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">--<span class="Apple-converted-space"> </span></span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Dave Täht</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">CEO, TekLibre, LLC</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><a href="http://www.teklibre.com/" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">http://www.teklibre.com</a><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Tel: 1-669-226-2619</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">_______________________________________________</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Cake mailing list</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><a href="mailto:Cake@lists.bufferbloat.net" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">Cake@lists.bufferbloat.net</a><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><a href="https://lists.bufferbloat.net/listinfo/cake" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">https://lists.bufferbloat.net/listinfo/cake</a></div></blockquote></div><br class=""></body></html>