[Bloat] Re: [Cake] Re: XDP2 is here - from one and only Tom Herbert (almost to the date, 10 years after XDP was released)

[Frantisek Borsik <frantisek.borsik@gmail.com>]

Fresh from Tom's oven:
https://medium.com/@tom_84912/programming-a-parser-in-xdp2-is-as-easy-as-pie-8f26c8b3e704


All the best,

Frank

Frantisek (Frank) Borsik


*In loving memory of Dave Täht: *1965-2025

https://libreqos.io/2025/04/01/in-loving-memory-of-dave/


https://www.linkedin.com/in/frantisekborsik

Signal, Telegram, WhatsApp: +421919416714

iMessage, mobile: +420775230885

Skype: casioa5302ca

frantisek.borsik@gmail.com


On Mon, Sep 15, 2025 at 8:35 PM Tom Herbert <tom@herbertland.com> wrote:

> On Mon, Sep 15, 2025 at 11:07 AM Frantisek Borsik
> <frantisek.borsik@gmail.com> wrote:
> >
> >
> > "There were a few NIC's that offloaded eBPF but they never really went
> mainstream."
> >
> > And even then, they were doing only 40 Gbps, like https://netronome.com
> and didn't even supported full eBPF...
> >
> > They only support a pretty small subset of eBPF (in particular they
> don't support the LPM map type, which was our biggest performance pain
> point), and have a pretty cool user replaceable firmware system. They also
> don't have the higher speeds - above 40 Gbps - where the offloading would
> be most useful."
>
> Yeah, the attempts at offloading eBPF were doomed to fail. It's a
> restricted model, lacks parallelism, doesn't support inline
> accelerators, and requires the eBPF VM to make it no-staters. DPDK
> would fail as well. The kernel/host environment and hardware
> environments are quite different. If we try to force the hardware to
> look like the host to make eBPF or DPDK portable then we'll lose the
> performance advantages of running in the hardware. We need a model
> that allows the software to adapt to HW, not the other way around (of
> course, in a perfect world we'd do software/hardware codesign from the
> get-go).
>
> >
> > Btw, Tom will be at FLOSS Weekly tomorrow (Tuesday), 12:20 EDT / 11:20
> CDT / 10:20 MDT / 9:20 PDT
>
> Can't wait!
>
> >
> > https://www.youtube.com/live/OBW5twvmHOI
> >
> >
> > All the best,
> >
> > Frank
> >
> > Frantisek (Frank) Borsik
> >
> >
> > In loving memory of Dave Täht: 1965-2025
> >
> > https://libreqos.io/2025/04/01/in-loving-memory-of-dave/
> >
> >
> > https://www.linkedin.com/in/frantisekborsik
> >
> > Signal, Telegram, WhatsApp: +421919416714
> >
> > iMessage, mobile: +420775230885
> >
> > Skype: casioa5302ca
> >
> > frantisek.borsik@gmail.com
> >
> >
> >
> > On Mon, Sep 15, 2025 at 5:16 PM Stephen Hemminger <
> stephen@networkplumber.org> wrote:
> >>
> >> On Mon, 15 Sep 2025 08:39:48 +0000
> >> BeckW--- via Bloat <bloat@lists.bufferbloat.net> wrote:
> >>
> >> > Programming networking hardware is a bit like programming 8 bit
> computers int the 1980s, the hardware is often too limited and varied to
> support useful abstractions. This is also true for CPU-based networking
> once you get into the >10 Gbps realm, when caching and pipelining
> architectures become relevant. Writing a network protocol compiler that
> produces efficient code for different NICs and different CPUs is a daunting
> task. And unlike with 8 bit computers, there are no simple metrics ('you
> need at least 32kb RAM to run this code' vs 'this NIC supports 4k queues
> with PIE, Codel', 'this CPU has 20 Mbyte of Intel SmartCache').
> >>
> >> Linux kernel still lacks an easy way to setup many features in Smart
> NIC's. DPDK has rte_flow which allows direct
> >> access to hardware flow processing. But DPDK lacks any reasonable form
> of shaper control.
> >>
> >> > Ebpf is very close to what was described in this 1995 exokernel
> paper(
> https://pdos.csail.mit.edu/6.828/2008/readings/engler95exokernel.pdf).
> The idea of the exokernel was to have easily loadable, verified code in the
> kernel -- eg the security-critical task of assigning a packet to a session
> of a user -- and leave the rest of the protocol -- eg tcp retransmissions
> -- to the user space. AFAIK few people use ebpf like this, but it should be
> possible.
> >> >
> >> > Ebpf manages the abstraction part well, but sacrifices a lot of
> performance -- eg lack of aggressive batching like vpp / fd.io does. With
> DPDK,  you often find out that your nic's hardware or driver doesn't
> support the function that you hoped to use and end up optimizing for a
> particular hardware. Even if driver and hardware support a functionality,
> it may very well be that hardware resources are too limited for your
> particular use case. The abstraction is there, but your code is still
> hardware specific.
> >>
> >> There were a few NIC's that offloaded eBPF but they never really went
> mainstream.
> >>
> >
> >
> >>
> >> > -----Ursprüngliche Nachricht-----
> >> > Von: David P. Reed <dpreed@deepplum.com>
> >> > Gesendet: Samstag, 13. September 2025 22:33
> >> > An: Tom Herbert <tom@herbertland.com>
> >> > Cc: Frantisek Borsik <frantisek.borsik@gmail.com>; Cake List <
> cake@lists.bufferbloat.net>; codel@lists.bufferbloat.net; bloat <
> bloat@lists.bufferbloat.net>; Jeremy Austin via Rpm <
> rpm@lists.bufferbloat.net>
> >> > Betreff: [Bloat] Re: [Cake] Re: XDP2 is here - from one and only Tom
> Herbert (almost to the date, 10 years after XDP was released)
> >> >
> >> >
> >> > Tom -
> >> >
> >> > An architecture-independent network framework independent of the OS
> kernel's peculiarities seems within reach (though a fair bit of work), and
> I think it would be a GOOD THING indeed. IMHO the Linux networking stack in
> the kernel is a horrific mess, and it doesn't have to be.
> >> >
> >> > The reason it doesn't have to be is that there should be no reason it
> cannot run in ring3/userland, just like DPDK. And it should be built using
> "real-time" userland programming techniques. (avoiding the generic linux
> scheduler). The ONLY reason for involving the scheduler would be because
> there aren't enough cores. Linux was designed to be a uniprocessor Unix,
> and that just is no longer true at all. With hyperthreading, too, one need
> never abandon a processor's context in userspace to run some "userland"
> application.
> >> >
> >> > This would rip a huge amount of kernel code out of the kernel. (at
> least 50%, and probably more). THe security issues of all those 3rd party
> network drivers would go away.
> >> >
> >> > And the performance would be much higher for networking.  (running in
> ring 3, especially if you don't do system calls, is no performance penalty,
> and interprocessor communications using shared memory is much lower latency
> than Linux IPC or mutexes).
> >> >
> >> > I like the idea of a compilation based network stack, at a slightly
> higher level than C. eBPF is NOT what I have in mind - it's an interpreter
> with high overhead. The language should support high-performance
> co-routining - shared memory, ideally. I don't thing GC is a good thing.
> Rust might be a good starting point because its memory management is safe.
> >> > To me, some of what the base of DPDK is like is good stuff. However,
> it isn't architecturally neutral.
> >> >
> >> > To me, the network stack should not be entangled with interrupt
> handling at all. "polling" is far more performant under load. The only use
> for interrupts is when the network stack is completely idle. That would be,
> in userland, a "wait for interrupt" call (not a poll). Ideally, on recent
> Intel machines, a userspace version of MONITOR/MWAIT).
> >> >
> >> > Now I know that Linus and his crew are really NOT gonna like this.
> Linus is still thinking like MINIX, a uniprocessor time-sharing system with
> rich OS functions in the kernel and doing "file" reads and writes to
> communicate with the kernel state. But it is a much more modern way to
> think of real-time IO in a modern operating system. (Windows and macOS are
> also Unix-like, uniprocessor monolithic kernel designs).
> >> >
> >> > So, if XDP2 got away from the Linux kernel, it could be great.
> >> > BTW, io_uring, etc. are half-measures. They address getting away from
> interrupts toward polling, but they still make the mistake of keeping huge
> drivers in the kernel.
> >>
> >> DPDK already supports use of XDP as a way to do userspace networking.
> >> It is good generic way to get packets in/out but the dedicated
> userspace drivers allow
> >> for more access to hardware. The XDP abstraction gets in the way of
> little things like programming
> >> VLAN's, etc.
> >>
> >> The tradeoff is userspace networking works great for infrastructure,
> routers, switches, firewalls etc;
> >> but userspace networking for network stacks to applications is hard to
> do, and loses the isolation
> >> that the kernel provides.
> >>
> >> >  > I think it is interesting as a concept. A project I am advising
> has been  > using DPDK very effectively to get rid of the huge path and
> locking delays  > in the current Linux network stack. XDP2 could be
> supported in a ring3  > (user) address space, achieving a similar result.
> >> > HI David,
> >> > The idea is you could write the code in XDP2 and it would be compiled
> to DPDK or eBPF and the compiler would handle the optimizations.
> >> >  >
> >> >  >
> >> >  >
> >> >  > But I don't think XDP2 is going that direction - so it may be
> stuckinto  > the mess of kernel space networking. Adding eBPF only has made
> this more of  > a mess, by the way (and adding a new "compiler" that needs
> to be veriried  > as safe for the kernel).
> >> > Think of XDP2 as the generalization of XDP to go beyond just the
> kernel. The idea is that the user writes their datapath code once and they
> compile it to run in whatever targets they have-- DPDK, P4, other
> programmable hardware, and yes XDP/eBPF. It's really not limited to kernel
> networking.
> >> > As for the name XDP2, when we created XDP, eXpress DataPath, my
> vision was that it would be implementation agnostic. eBPF was the first
> instantiation for practicality, but now ten years later I think we can
> realize the initial vision.
> >> > Tom
> >>
> >>
> >> At this point, different network architectures get focused at different
> use cases.
> >> The days of the one-size-fits-all networking of BSD Unix is dead.
>