[Bloat] high speed networking from userspace

[Bloat] high speed networking from userspace

[Luigi Rizzo]

Hi,
Dave mentioned me the thread about netmap on this list, to which
i just subscribed.

Some of the posts are referring to Van Jacobson's "network channels"
and to previous experiments or implementations dating back to 2006 e.g.

	http://www.ioremap.net/taxonomy/term/6

I am glad to see that there were previous attempts at addressing the
problem. However before dismissing things as 'done before'
i would suggest some performance comparison.

For netmap what i can offer at the moment is some data comparing
raw packet I/O performance with various sockets families,
libpcap, PACKET_TX_RING on linux, and even the in-kernel
packet generator in Linux. See for instance see a recent ACM Queue
paper (freely accessible), fig.4 and 5

        http://queue.acm.org/detail.cfm?id=2103536

In all these cases netmap is one order of magnitude or more
faster than the alternatives.

Unfortunately I cannot find any actual performance data on netchannel
except those from the LCA'06 Van Jacobson slides, where they show
about 2x speedup over the host stack. The original
netchannel site reports 404 on all links, e.g.

        http://tservice.net.ru/~s0mbre/blog/2006/10/26#2006_10_26

If someone had some performance data or examples of technologies
that work well i would be grateful to see them.

Comparing netmap with VJ network channels:
- both try to remove skbuf, and move processing out of the
  interrupt/kernel bottom half and into the user thread (above or
  below the userland/kernel barrier) to improve cache locality and
  for other good reasons.

  These two are key ideas for improving performance, which for instance
  PF_PACKET does not use (convenient as it does not need
  driver modifications, but there is a huge cost in performance.)

I cannot easily tell whether netchannel implements any of these features:
- userspace visible buffers (saves a memory copy and data access
  which may be helpful for packet forwarding apps where you
  only look at part of the payload)
- poll-able file descriptor (useful to build a pcap layer on
  top of the packet I/O framework)

cheers
luigi

[Bloat] high speed networking from userspace

[Dave Taht]

http://info.iet.unipi.it/~luigi/netmap/

is the by the same guy that did QFQ, and the results are quite
impressive. He (today) announced support for this interface for Linux.

shades of VJ's 'network channels'!

-- 
Dave Täht
SKYPE: davetaht
US Tel: 1-239-829-5608
http://www.bufferbloat.net

Re: [Bloat] high speed networking from userspace

[Ketan Kulkarni]

[-- Attachment #1: Type: text/plain, Size: 601 bytes --]

 will it help/impact bloat issue?

-Ketan

On Tue, Mar 13, 2012 at 5:39 AM, Dave Taht <dave.taht@gmail.com> wrote:

> http://info.iet.unipi.it/~luigi/netmap/
>
> is the by the same guy that did QFQ, and the results are quite
> impressive. He (today) announced support for this interface for Linux.
>
> shades of VJ's 'network channels'!
>
> --
> Dave Täht
> SKYPE: davetaht
> US Tel: 1-239-829-5608
> http://www.bufferbloat.net
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>

[-- Attachment #2: Type: text/html, Size: 1203 bytes --]

Re: [Bloat] high speed networking from userspace

[Steinar H. Gunderson]

On Mon, Mar 12, 2012 at 05:09:44PM -0700, Dave Taht wrote:
> http://info.iet.unipi.it/~luigi/netmap/
> 
> is the by the same guy that did QFQ, and the results are quite
> impressive. He (today) announced support for this interface for Linux.

How is this different from Linux' mmap packet socket?

/* Steinar */
-- 
Homepage: http://www.sesse.net/

Re: [Bloat] high speed networking from userspace

[Hagen Paul Pfeifer]

On Mon, 12 Mar 2012 17:09:44 -0700, Dave Taht <dave.taht@gmail.com> wrote:

> is the by the same guy that did QFQ, and the results are quite
> impressive. He (today) announced support for this interface for Linux.
> 
> shades of VJ's 'network channels'!

"already" implemented:

see

o http://www.ioremap.net/node/12
o http://www.ioremap.net/taxonomy/term/6

and all netdev discussions several years ago.

HGN

Re: [Bloat] high speed networking from userspace

[Stephen Hemminger]

On Tue, 13 Mar 2012 13:28:52 +0100
Hagen Paul Pfeifer <hagen@jauu.net> wrote:

> 
> On Mon, 12 Mar 2012 17:09:44 -0700, Dave Taht <dave.taht@gmail.com> wrote:
> 
> > is the by the same guy that did QFQ, and the results are quite
> > impressive. He (today) announced support for this interface for Linux.
> > 
> > shades of VJ's 'network channels'!
> 
> "already" implemented:
> 
> see
> 
> o http://www.ioremap.net/node/12
> o http://www.ioremap.net/taxonomy/term/6
> 
> and all netdev discussions several years ago.
> 
> HGN

User space networking works well for single application be it routing,
bridging, network trading, or single appliance. It doesn't work on a
multi-application environment (ie desktop). The gain is only because
the userspace code can choose to do less, but do it faster. So if you
want full stack, and firewall; don't bother.

Re: [Bloat] high speed networking from userspace

[Hagen Paul Pfeifer]

On Tue, 13 Mar 2012 09:03:57 -0700, Stephen Hemminger wrote:

> User space networking works well for single application be it routing,
> bridging, network trading, or single appliance. It doesn't work on a
> multi-application environment (ie desktop). The gain is only because
> the userspace code can choose to do less, but do it faster. So if you
> want full stack, and firewall; don't bother.

Thanks for the additional comments, that's why I wrote "see netdev
discussions". In sum: if you strip, disable, swap-out functionality you can
gain speed/latency - sounds like a universally valid statement ...

HGN

Re: [Bloat] high speed networking from userspace

[Luigi Rizzo]

On Tue, Mar 13, 2012 at 09:03:57AM -0700, Stephen Hemminger wrote:
> On Tue, 13 Mar 2012 13:28:52 +0100
> Hagen Paul Pfeifer <hagen@jauu.net> wrote:
> 
> > 
> > On Mon, 12 Mar 2012 17:09:44 -0700, Dave Taht <dave.taht@gmail.com> wrote:
> > 
> > > is the by the same guy that did QFQ, and the results are quite
> > > impressive. He (today) announced support for this interface for Linux.
> > > 
> > > shades of VJ's 'network channels'!
> > 
> > "already" implemented:
> > 
> > see
> > 
> > o http://www.ioremap.net/node/12
> > o http://www.ioremap.net/taxonomy/term/6
> > 
> > and all netdev discussions several years ago.
> > 
> > HGN
> 
> User space networking works well for single application be it routing,
> bridging, network trading, or single appliance. It doesn't work on a
> multi-application environment (ie desktop). The gain is only because
> the userspace code can choose to do less, but do it faster. So if you
> want full stack, and firewall; don't bother.

The firewall is actually one place where an efficient I/O mechanism
is really useful. Netmap (or the netfilter API or netgraph in
FreeBSD, if they were not built on top of skbufs/mbufs) give you
an ideal place to efficiently drop rogue traffic, and reinject the
interesting one in the stack for further processing.

Also the concepts used in netmap (and in VJ's netchannel)
are not confined to userspace networking.
Even in the kernel one can and probably should:

- get rid of skbufs/mbufs even in the kernel (replacing
  them with cheaper containers or data copies)
- do more packet coalescing (software RSC is an example), to amortize
  certain costs over larger batches;
- move work away from the interrupt/polling threads and closer 
  to the user thread (for better cache locality and load management)
  
The real gain of these mechanisms, i think, is having the option
to avoid costly operations when you don't need them.

That's the message i would like to convey.

Of course everything would be more convincing if i came up
with a full skbuf-less in-kernel stack and not just the
bottom layer+libpcap :)

cheers
luigi

Re: [Bloat] high speed networking from userspace

[Eric Dumazet]

On Tue, 2012-03-13 at 20:08 +0100, Luigi Rizzo wrote:

> The firewall is actually one place where an efficient I/O mechanism
> is really useful. Netmap (or the netfilter API or netgraph in
> FreeBSD, if they were not built on top of skbufs/mbufs) give you
> an ideal place to efficiently drop rogue traffic, and reinject the
> interesting one in the stack for further processing.
> 
> Also the concepts used in netmap (and in VJ's netchannel)
> are not confined to userspace networking.
> Even in the kernel one can and probably should:
> 
> - get rid of skbufs/mbufs even in the kernel (replacing
>   them with cheaper containers or data copies)
> - do more packet coalescing (software RSC is an example), to amortize
>   certain costs over larger batches;
> - move work away from the interrupt/polling threads and closer 
>   to the user thread (for better cache locality and load management)
>   
> The real gain of these mechanisms, i think, is having the option
> to avoid costly operations when you don't need them.
> 
> That's the message i would like to convey.
> 
> Of course everything would be more convincing if i came up
> with a full skbuf-less in-kernel stack and not just the
> bottom layer+libpcap :)
> 

OK, but what about process scheduler and ability to queue packets
somewhere if your low priority application is stalled because of some
high priority stuff coming, or what happens if your tcp receive windows
are 16Mbytes per flow...

Pre-allocating huge ring buffers is not an option if you handle thousand
of flows.

We could avoid memory allocators everywhere and come back to 30 years
old designs and MSDOS. But thats not the path taken by modern stuff.

netchannels have the multiplex/demux problem, and this need some
hardware support. Once you have decent hardware support, and xx core
machines, you can scale as you need with traditional stacks, as long as
you fully understand cache issues and memory locality.

Most problems come if you want to use NICs with one queue and one cpu.
This just doesnt make sense in 2012, does it ?

Sure, all sort of tricks can be used to implement full stack in user
land and be fast. Just make sure a NIC can be efficiently shared by this
application and others as well.

We had the opposite (implement a web server in kernel) and it was
probably an interesting idea in its time, but in the long term, you can
see nobody uses this anymore.