Re: [Bloat] high speed packet and protocol processing in userspace?

[Dave Taht <dave.taht@gmail.com>]

On Thu, Mar 16, 2017 at 10:32 AM, Michael Richardson <mcr@sandelman.ca> wrote:
>
> Dave Taht <dave.taht@gmail.com> wrote:
>     > I'm *really* impressed by the level of tool support emerging for ebpf,
>     > but it's still essentially writing assembly when I'd rather use an
>     > equivalent of bison or peg. If taking that massive step backwards is
>     > the right thing, to build that tool (if none exists)... well... ok.
>
> years ago, I worked in the NPU space, and we created the language that I
> think you wanted.  It was described here:
>
> https://www.ietf.org/archive/id/draft-nossik-pax-pdl-00.txt
>
> IPv4 patterns filters looked like this:
>
> PATTERN  IP_v4_Hdr_For_ICMP {
>   version UINT 4 == 4;
>   ihl UINT 4 == 5;            /* length == 5 : no options */
>   typeOfService UINT 8;
>   totalLength UINT 16;
>   identification UINT 16;
>   flagReserved BIT 1 == 0;
>   flagDontFragment BIT 1;
>   flagMoreFragments BIT 1 == 0;    /* last fragments only */
>   fragmentOffset UINT 13 == 0;    /* first fragments only */
>   timeToLive UINT 8;
>   protocol BIT 8 == IPPROTO_ICMP;                 /* ICMP */
>   headerChecksum BIT 16;
>   sourceAddress BIT 32;
>   destinationAddress BIT 32
> }
>
> You could compile the patterns, and then instantiate fields at runtime if you
> needed to.  (i.e: add constraints).  Also merge patterns to select (a) or (b)
> in constant time (but not constant memory usage)

Yes, that is more or less what I wanted. And in 1998!?

> Solidum (for whom I worked at the time) had a compiler that would turn this
> into ) patricia-tree filters that ran in hardware at wirespeed.  But, APIs
> into ernels were hard to get accepted.
> In the end, only people willing to pay for this were TLAs who wanted to run
> regex's on traffic. Regex took lots of ram, but TLAs didn't buy many sockets.

I think regexes are flexible but crazy for high speed packet
processing of formally defined protocols that essentially only make
forward progress.

the chip design people must have had tools like this for decades now,
but not OS and app design folk. :sigh:

> I would love to be able to recycle the compiler as open source; it could
> easily spit out EBPF, but generating assembly directly might be faster
> actually.  The Patricia tree is basically jmp base+nextbyteofpacket., but
> stored more compactly.

Well, ebpf as a target would make it most useful at the moment,
followed by pure C (processing a packet), followed by verilog. Still
have the code?

I'm tempted to take apart the first tool mentioned to see if it could
be made to (in addition to verilog) output c, then bpf, but I had
really rather expected this to have been a solved problem. And it
seemed to have been! in 1998! wow

... but, like everyone else, I go around painfully writing custom pcap
filters for everything. It's begun to really, really hurt when I do
that.

I need to also note that I don't care about tcp - we have plenty of
tools for that -  in working with the airtime-pie-chart I discovered
it was filtering out all the multicast beacons, and all the other
related "below layer 1" wifi traffic, probes, crypto etc - in addition
to the failures in nd and dhcp and babel I was looking at.

the amount of that stuff was significant - 12% typically where I am,
I've seen +40% - and now that I know where to look, perhaps I'll find
more, and why....

And I'd been eyeball-filtering it out for a long time when looking at aircaps.

And regrettably very little - except the IEEE specifications - defines
what the correct behaviors for all that is.

So I turned over a rock and found a nightmare in wifi I hadn't deeply suspected.


> --
> ]               Never tell me the odds!                 | ipv6 mesh networks [
> ]   Michael Richardson, Sandelman Software Works        | network architect  [
> ]     mcr@sandelman.ca  http://www.sandelman.ca/        |   ruby on rails    [
>
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>



-- 
Dave Täht
Let's go make home routers and wifi faster! With better software!
http://blog.cerowrt.org