[Bloat] recommended PC config for network testing using Ubuntu

Mon Feb 23 13:31:21 EST 2015

I do confess to being disappointed that the cisco ethernet drivers in the
linux kernel have not been updated to use BQL. It takes a couple hours to
code up the 2-6 lines of additional code needed, if you have the hardware.
The core thing to look for is any hardware that has bql support for
ethernet.

http://www.bufferbloat.net/projects/bloat/wiki/BQL_enabled_drivers

(this list is a bit behind, I think 3 or more new BQL drivers got added in
the last two kernel revs, including the TI 10gig chip and I forget the
other two)

For rack mounts presently bufferbloat.net mostly uses a bunch of donated
cast off hardware, held together with chewing gum and bailing wire. More
donations gladly appreciated.

Back in the day I used to buy a lot of gear from:

http://www.penguincomputing.com/products/rackmount-servers/relion-servers/

And then there is hp, dell, etc, which do linux boxes fairly well in the dc
space and ship with modern kernels.

But these days I mostly build my own, usually around an asus motherboard.
There is a ton of work around the opencompute stuff:
http://www.opencompute.org/projects/server/ but most of it is so much
higher end than is needed to forward packets at the rates I care about - IF
I wanted to do 160Gps or more I would be fiddling with these 12 and 16 core
xeon boxes, but I am not....

In trying to stay low cost, fan free...

I went through hell trying out the supermicro rangeley platform - the
manufacturer I tried was under the delusion that an 8 core rangeley did not
need a fan, in particular, and nobody at the time was making a rackmount
case with the right sort of power supply for it nor port breakouts, and I
never found a way to reliabily boot from and use a USB stick for more than
a few days, so I ended up going back to a normal sata drive for it, and
going through a half dozen power supplies and cases. I do hope someone is
now packaging up that sort of box properly now, in a form that can be used.
If anyone wants 4 useless rack mount cases I never got around to shipping
them back.

Once I got a rangeley running it is  *excellent* at forwarding packets in
aqm/fq algos at GigE line rate (due to the ivy bridge dma to cache
architecture) but not very good at driving tests directly (due to the still
quite weak cpu cores), but it does do effective software rate control + an
aqm/fq algorithm at GigE speeds. Also the ethernet chips in it are well
supported by BQL and the new xmit_more stuff. A MAJOR testing issue is that
the ethernet driver has 8 hardware queues and unless you are careful/aware
of that, or disable you run into birthday problems everywhere and results
that dont make sense. (in the general case, running without the 8 queues
proves faster with fq_codel than with the 8)

It is very nice to have 4 onboard ethernet ports also, which makes testing
a variety of scenarios (like 2 ports into one) a snap. I never got around
to trying to make it do 10GigE although the cards for that are lying around
here somewhere.

I used intel i3 based NUCs as my primary test drivers for the past year.
Small, silent, fast. Hardware offloads are needed to drive them to gigE, so
they are a little wrong for direcly evaluating AQM/fq - but good if you
want to make sure those algos work with TSO/GSO/GRO correctly. I218-v
ethernet. Cant route unless you want to observe the horrors that are in
present day usb to ethernet adaptors.

For laptops, I am generally sticking with the older lenovo ones off-lease
and off of craigslist that had a decent keyboard, unlike the chicklet stuff
they are now shipping. I swore by the old T60 and T61s and have T400 and
T440 ones now. Lenovo gear generally still, even the chicklet ones, have
very good linux support (and are probably the most common laptop you will
see in the linux community). E1000E ethernet, good support for bql and
xmit_more.

The chromebooks from HP, with the ath9k in them, can actually be turned
into a decent test platform once you replace the OS on them. I am actually
going to go get another one today (my last two were stolen) as we are
making really good progress on improving wifi behavior of late and I need
more client gear to test with.  They dont have ethernet.

As for a desktop I am terribly pleased with snapon - no crashes *ever* in
3+ years running, and it cost about 2.2k to build, and if I could remember
the parts in it, I would build another one just like it - 64GB ram i7
6-core, flash disk, liquid cooling and all.

As for a home router, on openwrt chaos calmer everything that is ath9k and
ar71xx based is reasonable at rates below 60mbit for soft rate shaping and
I have had good results of late without at 500+mbit hard (I dont see aqm
engaging but I do see fq working). Tons of other chipsets, YMMV.

My topmost candidate for a cheap home router to work with going forward is
the tp-link archer c7 v2 although that might change if I can find the
DIR-650L model B somewhere. The archer has a pathetic cpu but both ath9k
and ath10k chips, and my focus is more on fixing wifi than fixing ethernet
these days.

The archer´s ath10k was AWEFUL when I last tried it but I think that can
now be improved

http://snapon.lab.bufferbloat.net/~d/archer/overnight/normality2.png

Still not a lot of hope for the topmost netgear (x4), asus, and dp-link
models as yet. IMHO QCA is doing the best job of getting their latest cpu
stuff into the linux mainline, but all these architectures use proprietary
hardware offloads that are impossible to get fingers into and are not
mainlinable. Best we can hope to see is that their chipset vendors are
taking the openly available algorithms and burning them into their
proprietary firmware. I have certainly talked to them enough about it (and
am not in a position to say who is doing what, sorry - but although I am
encouraged by the progress behind the scenes, it is taking way too long for
these guys to implement an algorithm eric dumazet wrote and mainlined in a
single saturday afternoon). Proprietary firmware is not particularly
helpful for researchers, I know...

I do wish very much I could find a low cost platform that could forward
without offloads at line rate with aqm/fq AND do software rate limiting
+aqm/fq to nearly gigE on both inbound and outbound, but so far, have not
found one. The AMD cougar products cant crack 600mbits, you can almost but
not quite get there on an intel 3, and I have not surveyed the rangeley
market in the 6 months since I got the last one to work.

The older (and commonly available with two ethernet ports) atoms *sucked*:

http://snapon.lab.bufferbloat.net/~d/Native_GigE_Atoms_NoOffloads-5873/

I do have several 64bit 8 arm products under evaluation, they barely boot.
The Xgene I can probably talk about now, but I havent tried to boot it in a
while. The new TI stuff was looking good, havent got around to it. the
current crop of cheap boards for arm a8, a9, etc, equipped with ethernet
have really lousy drivers in them. But ooh! 4 cores! Shiny!

If you would like a specific recommendation for a specific rackmount in
particular, I need to go get a couple for an upcoming round of testing, and
as you can tell, just as confused about what to buy as anyone else. I am
*very* sure about which ethernet chips are worth buying, pretty sure that I
can get away with rangeley for gigE routing, dubious about arm, angry at
offloaders, that is about it.

On Mon, Feb 23, 2015 at 7:14 AM, Bill Ver Steeg (versteb) <versteb at cisco.com
> wrote:

>  I have been running network tests for several years on a mix of older
> Cisco UCS servers and old HP desktops.
>
>
>
> It seems that releases of the vintage of 3.2.0.18 run fine on these
> systems, but when I try to move forward to 3.19.0 I seem to bump into
> compatibility problems. Rather than spend a bunch of time resolving my
> configuration issues, I thought I would ask this illustrious group what
> PC-based platforms they are using to run the latest AQM code. Hopefully I
> can just pick up a new server or two and be back in business.
>
>
>
> Thanks in advance.
>
>
>
> Bvs
>
>
>
>
>
> [image: http://www.cisco.com/web/europe/images/email/signature/logo05.jpg]
>
> *Bill Ver Steeg*
> DISTINGUISHED ENGINEER
> versteb at cisco.com
>
>
>
>
>
>
>
>
>
>
>
> _______________________________________________
> Bloat mailing list
> Bloat at lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>
>

-- 
Dave Täht

thttp://www.bufferbloat.net/projects/bloat/wiki/Upcoming_Talks
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.bufferbloat.net/pipermail/bloat/attachments/20150223/9ad1a04f/attachment-0003.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.jpg
Type: image/jpeg
Size: 2527 bytes
Desc: not available
URL: <https://lists.bufferbloat.net/pipermail/bloat/attachments/20150223/9ad1a04f/attachment-0003.jpg>