<div dir="ltr"><div>I do confess to being disappointed that the cisco ethernet drivers in the linux kernel have not been updated to use BQL. It takes a couple hours to code up the 2-6 lines of additional code needed, if you have the hardware. The core thing to look for is any hardware that has bql support for ethernet.</div><div><br></div><div><a href="http://www.bufferbloat.net/projects/bloat/wiki/BQL_enabled_drivers">http://www.bufferbloat.net/projects/bloat/wiki/BQL_enabled_drivers</a><br></div><div><br></div><div>(this list is a bit behind, I think 3 or more new BQL drivers got added in the last two kernel revs, including the TI 10gig chip and I forget the other two)</div><div><br></div>For rack mounts presently <a href="http://bufferbloat.net">bufferbloat.net</a> mostly uses a bunch of donated cast off hardware, held together with chewing gum and bailing wire. More donations gladly appreciated.<div><br></div><div>Back in the day I used to buy a lot of gear from:</div><div><br></div><div><a href="http://www.penguincomputing.com/products/rackmount-servers/relion-servers/">http://www.penguincomputing.com/products/rackmount-servers/relion-servers/</a><br></div><div><br></div><div>And then there is hp, dell, etc, which do linux boxes fairly well in the dc space and ship with modern kernels.</div><div><br></div><div>But these days I mostly build my own, usually around an asus motherboard. There is a ton of work around the opencompute stuff: <a href="http://www.opencompute.org/projects/server/">http://www.opencompute.org/projects/server/</a> but most of it is so much higher end than is needed to forward packets at the rates I care about - IF I wanted to do 160Gps or more I would be fiddling with these 12 and 16 core xeon boxes, but I am not.... </div><div><br></div><div>In trying to stay low cost, fan free...</div><div><br></div><div>I went through hell trying out the supermicro rangeley platform - the manufacturer I tried was under the delusion that an 8 core rangeley did not need a fan, in particular, and nobody at the time was making a rackmount case with the right sort of power supply for it nor port breakouts, and I never found a way to reliabily boot from and use a USB stick for more than a few days, so I ended up going back to a normal sata drive for it, and going through a half dozen power supplies and cases. I do hope someone is now packaging up that sort of box properly now, in a form that can be used. If anyone wants 4 useless rack mount cases I never got around to shipping them back.</div><div><br></div><div>Once I got a rangeley running it is *excellent* at forwarding packets in aqm/fq algos at GigE line rate (due to the ivy bridge dma to cache architecture) but not very good at driving tests directly (due to the still quite weak cpu cores), but it does do effective software rate control + an aqm/fq algorithm at GigE speeds. Also the ethernet chips in it are well supported by BQL and the new xmit_more stuff. A MAJOR testing issue is that the ethernet driver has 8 hardware queues and unless you are careful/aware of that, or disable you run into birthday problems everywhere and results that dont make sense. (in the general case, running without the 8 queues proves faster with fq_codel than with the 8)</div><div><br></div><div>It is very nice to have 4 onboard ethernet ports also, which makes testing a variety of scenarios (like 2 ports into one) a snap. I never got around to trying to make it do 10GigE although the cards for that are lying around here somewhere.</div><div><br></div><div>I used intel i3 based NUCs as my primary test drivers for the past year. Small, silent, fast. Hardware offloads are needed to drive them to gigE, so they are a little wrong for direcly evaluating AQM/fq - but good if you want to make sure those algos work with TSO/GSO/GRO correctly. I218-v ethernet. Cant route unless you want to observe the horrors that are in present day usb to ethernet adaptors.</div><div><br></div><div>For laptops, I am generally sticking with the older lenovo ones off-lease and off of craigslist that had a decent keyboard, unlike the chicklet stuff they are now shipping. I swore by the old T60 and T61s and have T400 and T440 ones now. Lenovo gear generally still, even the chicklet ones, have very good linux support (and are probably the most common laptop you will see in the linux community). E1000E ethernet, good support for bql and xmit_more.</div><div><br></div><div>The chromebooks from HP, with the ath9k in them, can actually be turned into a decent test platform once you replace the OS on them. I am actually going to go get another one today (my last two were stolen) as we are making really good progress on improving wifi behavior of late and I need more client gear to test with. They dont have ethernet.</div><div><br></div><div>As for a desktop I am terribly pleased with snapon - no crashes *ever* in 3+ years running, and it cost about 2.2k to build, and if I could remember the parts in it, I would build another one just like it - 64GB ram i7 6-core, flash disk, liquid cooling and all.</div><div><br></div><div>As for a home router, on openwrt chaos calmer everything that is ath9k and ar71xx based is reasonable at rates below 60mbit for soft rate shaping and I have had good results of late without at 500+mbit hard (I dont see aqm engaging but I do see fq working). Tons of other chipsets, YMMV. </div><div><br></div><div>My topmost candidate for a cheap home router to work with going forward is the tp-link archer c7 v2 although that might change if I can find the DIR-650L model B somewhere. The archer has a pathetic cpu but both ath9k and ath10k chips, and my focus is more on fixing wifi than fixing ethernet these days.</div><div><br></div><div>The archer´s ath10k was AWEFUL when I last tried it but I think that can now be improved</div><div><br></div><div><a href="http://snapon.lab.bufferbloat.net/~d/archer/overnight/normality2.png">http://snapon.lab.bufferbloat.net/~d/archer/overnight/normality2.png</a><br></div><div><br></div><div>Still not a lot of hope for the topmost netgear (x4), asus, and dp-link models as yet. IMHO QCA is doing the best job of getting their latest cpu stuff into the linux mainline, but all these architectures use proprietary hardware offloads that are impossible to get fingers into and are not mainlinable. Best we can hope to see is that their chipset vendors are taking the openly available algorithms and burning them into their proprietary firmware. I have certainly talked to them enough about it (and am not in a position to say who is doing what, sorry - but although I am encouraged by the progress behind the scenes, it is taking way too long for these guys to implement an algorithm eric dumazet wrote and mainlined in a single saturday afternoon). Proprietary firmware is not particularly helpful for researchers, I know...</div><div><br></div><div>I do wish very much I could find a low cost platform that could forward without offloads at line rate with aqm/fq AND do software rate limiting +aqm/fq to nearly gigE on both inbound and outbound, but so far, have not found one. The AMD cougar products cant crack 600mbits, you can almost but not quite get there on an intel 3, and I have not surveyed the rangeley market in the 6 months since I got the last one to work. </div><div><br></div><div>The older (and commonly available with two ethernet ports) atoms *sucked*:</div><div><br></div><div><a href="http://snapon.lab.bufferbloat.net/~d/Native_GigE_Atoms_NoOffloads-5873/">http://snapon.lab.bufferbloat.net/~d/Native_GigE_Atoms_NoOffloads-5873/</a><br></div><div><br></div><div>I do have several 64bit 8 arm products under evaluation, they barely boot. The Xgene I can probably talk about now, but I havent tried to boot it in a while. The new TI stuff was looking good, havent got around to it. the current crop of cheap boards for arm a8, a9, etc, equipped with ethernet have really lousy drivers in them. But ooh! 4 cores! Shiny!</div><div><br></div><div>If you would like a specific recommendation for a specific rackmount in particular, I need to go get a couple for an upcoming round of testing, and as you can tell, just as confused about what to buy as anyone else. I am *very* sure about which ethernet chips are worth buying, pretty sure that I can get away with rangeley for gigE routing, dubious about arm, angry at offloaders, that is about it. </div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Feb 23, 2015 at 7:14 AM, Bill Ver Steeg (versteb) <span dir="ltr"><<a href="mailto:versteb@cisco.com" target="_blank">versteb@cisco.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple">
<div>
<p class="MsoNormal">I have been running network tests for several years on a mix of older Cisco UCS servers and old HP desktops.
<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">It seems that releases of the vintage of 3.2.0.18 run fine on these systems, but when I try to move forward to 3.19.0 I seem to bump into compatibility problems. Rather than spend a bunch of time resolving my configuration issues, I thought
I would ask this illustrious group what PC-based platforms they are using to run the latest AQM code. Hopefully I can just pick up a new server or two and be back in business.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Thanks in advance.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Bvs<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<table border="0" cellspacing="0" cellpadding="0" width="815" style="width:407.25pt">
<tbody>
<tr>
<td colspan="3" style="padding:0in 0in 0in 0in">
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman","serif""><img width="165" height="110" src="cid:image003.jpg@01D04F51.7AF1DA00" alt="http://www.cisco.com/web/europe/images/email/signature/logo05.jpg"><u></u><u></u></span></p>
</td>
</tr>
<tr>
<td nowrap valign="top" style="padding:0in 0in 0in .25in">
<p class="MsoNormal"><b><span style="font-size:8.5pt;font-family:"Arial","sans-serif";color:#666666">Bill Ver Steeg</span></b><span style="font-size:8.5pt;font-family:"Arial","sans-serif";color:#666666"><br>
DISTINGUISHED ENGINEER <br>
<a href="mailto:versteb@cisco.com" target="_blank">versteb@cisco.com</a><br>
<br>
<u></u><u></u></span></p>
</td>
<td width="50%" valign="top" style="width:50.0%;padding:11.25pt 0in 7.5pt 15.0pt">
</td>
<td style="padding:0in 0in 0in 0in"></td>
</tr>
<tr>
<td colspan="3" style="padding:0in 0in 0in 0in">
<table border="0" cellspacing="0" cellpadding="0" width="100%" style="width:100.0%">
<tbody>
<tr>
<td style="padding:0in 15.0pt 3.75pt .25in"></td>
</tr>
<tr>
<td style="border:none;border-bottom:solid #cccccc 1.0pt;padding:0in 0in 0in 0in">
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:12.0pt;font-family:"Times New Roman","serif""><u></u> <u></u></span></p>
<table border="0" cellspacing="0" cellpadding="0" width="600" style="width:300.0pt">
<tbody>
<tr>
<td style="padding:0in 15.0pt 0in .25in">
<p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Arial","sans-serif";color:#009900"><u></u> <u></u></span></p>
</td>
</tr>
<tr>
<td style="padding:5.25pt 15.0pt 4.5pt .25in">
<p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Arial","sans-serif";color:#999999"><u></u> <u></u></span></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal"><span lang="EN-GB"><u></u> <u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
<br>_______________________________________________<br>
Bloat mailing list<br>
<a href="mailto:Bloat@lists.bufferbloat.net">Bloat@lists.bufferbloat.net</a><br>
<a href="https://lists.bufferbloat.net/listinfo/bloat" target="_blank">https://lists.bufferbloat.net/listinfo/bloat</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature">Dave Täht<br><br>thttp://<a href="http://www.bufferbloat.net/projects/bloat/wiki/Upcoming_Talks" target="_blank">www.bufferbloat.net/projects/bloat/wiki/Upcoming_Talks</a></div>
</div>