[Bloat] Re: [Cake] Re: XDP2 is here - from one and only Tom Herbert (almost to the date, 10 years after XDP was released)

[Tom Herbert <tom@herbertland.com>]

On Mon, Sep 15, 2025, 4:16 PM David P. Reed <dpreed@deepplum.com> wrote:

>
>
> On Monday, September 15, 2025 18:26, "Frantisek Borsik" <
> frantisek.borsik@gmail.com> said:
>
> > Fresh from Tom's oven:
> >
> https://medium.com/@tom_84912/programming-a-parser-in-xdp2-is-as-easy-as-pie-8f26c8b3e704
>
> Nice enough. I've always wondered why no one did a table driven packet
> parser. Surely wireshark did this?
>
> However, the spaghetti mess in practice partially arises from skbuff
> (Linux) or mbuf (unix) structuring.  I can testify to this because I've had
> to add a low level protocol on top of Ethernet in a version of the FreeBSD
> kernel, and that protocol had to (for performance reasons) modify headers
> "in place" to generate response packets without copying or reallocating.
>
> This is an additional reason why Linux's network stack is a mess - skbuff
> management is generally not compatible with packet formats in NICs with
> acceleration features that parse or verify parts of the packets.
>
> Admittedly, I've focused on minimizing end-to-end latency in protocols
> where the minimum speed is 10 Gb/sec Ethernet, where the hardware
> functionality (including pipelining the packet processing in the NIC
> controller, possibly with FPGAs, are essential.
>
> While the header parsing is a chunk of the performance issue, specialized
> memory management is also.
>
> Hoping to see what replaces skbuff in EDK2.
>

PVbufs :-)

https://medium.com/@tom_84912/pvbufs-hardware-friendly-metadata-structure-for-scatter-gather-6fce8d42f75f


> >
> >
> > All the best,
> >
> > Frank
> >
> > Frantisek (Frank) Borsik
> >
> >
> > *In loving memory of Dave Täht: *1965-2025
> >
> > https://libreqos.io/2025/04/01/in-loving-memory-of-dave/
> >
> >
> > https://www.linkedin.com/in/frantisekborsik
> >
> > Signal, Telegram, WhatsApp: +421919416714
> >
> > iMessage, mobile: +420775230885
> >
> > Skype: casioa5302ca
> >
> > frantisek.borsik@gmail.com
> >
> >
> > On Mon, Sep 15, 2025 at 8:35 PM Tom Herbert <tom@herbertland.com> wrote:
> >
> >> On Mon, Sep 15, 2025 at 11:07 AM Frantisek Borsik
> >> <frantisek.borsik@gmail.com> wrote:
> >> >
> >> >
> >> > "There were a few NIC's that offloaded eBPF but they never really went
> >> mainstream."
> >> >
> >> > And even then, they were doing only 40 Gbps, like
> https://netronome.com
> >> and didn't even supported full eBPF...
> >> >
> >> > They only support a pretty small subset of eBPF (in particular they
> >> don't support the LPM map type, which was our biggest performance pain
> >> point), and have a pretty cool user replaceable firmware system. They
> also
> >> don't have the higher speeds - above 40 Gbps - where the offloading
> would
> >> be most useful."
> >>
> >> Yeah, the attempts at offloading eBPF were doomed to fail. It's a
> >> restricted model, lacks parallelism, doesn't support inline
> >> accelerators, and requires the eBPF VM to make it no-staters. DPDK
> >> would fail as well. The kernel/host environment and hardware
> >> environments are quite different. If we try to force the hardware to
> >> look like the host to make eBPF or DPDK portable then we'll lose the
> >> performance advantages of running in the hardware. We need a model
> >> that allows the software to adapt to HW, not the other way around (of
> >> course, in a perfect world we'd do software/hardware codesign from the
> >> get-go).
> >>
> >> >
> >> > Btw, Tom will be at FLOSS Weekly tomorrow (Tuesday), 12:20 EDT / 11:20
> >> CDT / 10:20 MDT / 9:20 PDT
> >>
> >> Can't wait!
> >>
> >> >
> >> > https://www.youtube.com/live/OBW5twvmHOI
> >> >
> >> >
> >> > All the best,
> >> >
> >> > Frank
> >> >
> >> > Frantisek (Frank) Borsik
> >> >
> >> >
> >> > In loving memory of Dave Täht: 1965-2025
> >> >
> >> > https://libreqos.io/2025/04/01/in-loving-memory-of-dave/
> >> >
> >> >
> >> > https://www.linkedin.com/in/frantisekborsik
> >> >
> >> > Signal, Telegram, WhatsApp: +421919416714
> >> >
> >> > iMessage, mobile: +420775230885
> >> >
> >> > Skype: casioa5302ca
> >> >
> >> > frantisek.borsik@gmail.com
> >> >
> >> >
> >> >
> >> > On Mon, Sep 15, 2025 at 5:16 PM Stephen Hemminger <
> >> stephen@networkplumber.org> wrote:
> >> >>
> >> >> On Mon, 15 Sep 2025 08:39:48 +0000
> >> >> BeckW--- via Bloat <bloat@lists.bufferbloat.net> wrote:
> >> >>
> >> >> > Programming networking hardware is a bit like programming 8 bit
> >> computers int the 1980s, the hardware is often too limited and varied to
> >> support useful abstractions. This is also true for CPU-based networking
> >> once you get into the >10 Gbps realm, when caching and pipelining
> >> architectures become relevant. Writing a network protocol compiler that
> >> produces efficient code for different NICs and different CPUs is a
> daunting
> >> task. And unlike with 8 bit computers, there are no simple metrics ('you
> >> need at least 32kb RAM to run this code' vs 'this NIC supports 4k queues
> >> with PIE, Codel', 'this CPU has 20 Mbyte of Intel SmartCache').
> >> >>
> >> >> Linux kernel still lacks an easy way to setup many features in Smart
> >> NIC's. DPDK has rte_flow which allows direct
> >> >> access to hardware flow processing. But DPDK lacks any reasonable
> form
> >> of shaper control.
> >> >>
> >> >> > Ebpf is very close to what was described in this 1995 exokernel
> >> paper(
> >> https://pdos.csail.mit.edu/6.828/2008/readings/engler95exokernel.pdf).
> >> The idea of the exokernel was to have easily loadable, verified code in
> the
> >> kernel -- eg the security-critical task of assigning a packet to a
> session
> >> of a user -- and leave the rest of the protocol -- eg tcp
> retransmissions
> >> -- to the user space. AFAIK few people use ebpf like this, but it
> should be
> >> possible.
> >> >> >
> >> >> > Ebpf manages the abstraction part well, but sacrifices a lot of
> >> performance -- eg lack of aggressive batching like vpp / fd.io does.
> With
> >> DPDK,  you often find out that your nic's hardware or driver doesn't
> >> support the function that you hoped to use and end up optimizing for a
> >> particular hardware. Even if driver and hardware support a
> functionality,
> >> it may very well be that hardware resources are too limited for your
> >> particular use case. The abstraction is there, but your code is still
> >> hardware specific.
> >> >>
> >> >> There were a few NIC's that offloaded eBPF but they never really went
> >> mainstream.
> >> >>
> >> >
> >> >
> >> >>
> >> >> > -----Ursprüngliche Nachricht-----
> >> >> > Von: David P. Reed <dpreed@deepplum.com>
> >> >> > Gesendet: Samstag, 13. September 2025 22:33
> >> >> > An: Tom Herbert <tom@herbertland.com>
> >> >> > Cc: Frantisek Borsik <frantisek.borsik@gmail.com>; Cake List <
> >> cake@lists.bufferbloat.net>; codel@lists.bufferbloat.net; bloat <
> >> bloat@lists.bufferbloat.net>; Jeremy Austin via Rpm <
> >> rpm@lists.bufferbloat.net>
> >> >> > Betreff: [Bloat] Re: [Cake] Re: XDP2 is here - from one and only
> Tom
> >> Herbert (almost to the date, 10 years after XDP was released)
> >> >> >
> >> >> >
> >> >> > Tom -
> >> >> >
> >> >> > An architecture-independent network framework independent of the OS
> >> kernel's peculiarities seems within reach (though a fair bit of work),
> and
> >> I think it would be a GOOD THING indeed. IMHO the Linux networking
> stack in
> >> the kernel is a horrific mess, and it doesn't have to be.
> >> >> >
> >> >> > The reason it doesn't have to be is that there should be no reason
> it
> >> cannot run in ring3/userland, just like DPDK. And it should be built
> using
> >> "real-time" userland programming techniques. (avoiding the generic linux
> >> scheduler). The ONLY reason for involving the scheduler would be because
> >> there aren't enough cores. Linux was designed to be a uniprocessor Unix,
> >> and that just is no longer true at all. With hyperthreading, too, one
> need
> >> never abandon a processor's context in userspace to run some "userland"
> >> application.
> >> >> >
> >> >> > This would rip a huge amount of kernel code out of the kernel. (at
> >> least 50%, and probably more). THe security issues of all those 3rd
> party
> >> network drivers would go away.
> >> >> >
> >> >> > And the performance would be much higher for networking.  (running
> in
> >> ring 3, especially if you don't do system calls, is no performance
> penalty,
> >> and interprocessor communications using shared memory is much lower
> latency
> >> than Linux IPC or mutexes).
> >> >> >
> >> >> > I like the idea of a compilation based network stack, at a slightly
> >> higher level than C. eBPF is NOT what I have in mind - it's an
> interpreter
> >> with high overhead. The language should support high-performance
> >> co-routining - shared memory, ideally. I don't thing GC is a good thing.
> >> Rust might be a good starting point because its memory management is
> safe.
> >> >> > To me, some of what the base of DPDK is like is good stuff.
> However,
> >> it isn't architecturally neutral.
> >> >> >
> >> >> > To me, the network stack should not be entangled with interrupt
> >> handling at all. "polling" is far more performant under load. The only
> use
> >> for interrupts is when the network stack is completely idle. That would
> be,
> >> in userland, a "wait for interrupt" call (not a poll). Ideally, on
> recent
> >> Intel machines, a userspace version of MONITOR/MWAIT).
> >> >> >
> >> >> > Now I know that Linus and his crew are really NOT gonna like this.
> >> Linus is still thinking like MINIX, a uniprocessor time-sharing system
> with
> >> rich OS functions in the kernel and doing "file" reads and writes to
> >> communicate with the kernel state. But it is a much more modern way to
> >> think of real-time IO in a modern operating system. (Windows and macOS
> are
> >> also Unix-like, uniprocessor monolithic kernel designs).
> >> >> >
> >> >> > So, if XDP2 got away from the Linux kernel, it could be great.
> >> >> > BTW, io_uring, etc. are half-measures. They address getting away
> from
> >> interrupts toward polling, but they still make the mistake of keeping
> huge
> >> drivers in the kernel.
> >> >>
> >> >> DPDK already supports use of XDP as a way to do userspace networking.
> >> >> It is good generic way to get packets in/out but the dedicated
> >> userspace drivers allow
> >> >> for more access to hardware. The XDP abstraction gets in the way of
> >> little things like programming
> >> >> VLAN's, etc.
> >> >>
> >> >> The tradeoff is userspace networking works great for infrastructure,
> >> routers, switches, firewalls etc;
> >> >> but userspace networking for network stacks to applications is hard
> to
> >> do, and loses the isolation
> >> >> that the kernel provides.
> >> >>
> >> >> >  > I think it is interesting as a concept. A project I am advising
> >> has been  > using DPDK very effectively to get rid of the huge path and
> >> locking delays  > in the current Linux network stack. XDP2 could be
> >> supported in a ring3  > (user) address space, achieving a similar
> result.
> >> >> > HI David,
> >> >> > The idea is you could write the code in XDP2 and it would be
> compiled
> >> to DPDK or eBPF and the compiler would handle the optimizations.
> >> >> >  >
> >> >> >  >
> >> >> >  >
> >> >> >  > But I don't think XDP2 is going that direction - so it may be
> >> stuckinto  > the mess of kernel space networking. Adding eBPF only has
> made
> >> this more of  > a mess, by the way (and adding a new "compiler" that
> needs
> >> to be veriried  > as safe for the kernel).
> >> >> > Think of XDP2 as the generalization of XDP to go beyond just the
> >> kernel. The idea is that the user writes their datapath code once and
> they
> >> compile it to run in whatever targets they have-- DPDK, P4, other
> >> programmable hardware, and yes XDP/eBPF. It's really not limited to
> kernel
> >> networking.
> >> >> > As for the name XDP2, when we created XDP, eXpress DataPath, my
> >> vision was that it would be implementation agnostic. eBPF was the first
> >> instantiation for practicality, but now ten years later I think we can
> >> realize the initial vision.
> >> >> > Tom
> >> >>
> >> >>
> >> >> At this point, different network architectures get focused at
> different
> >> use cases.
> >> >> The days of the one-size-fits-all networking of BSD Unix is dead.
> >>
> >
>
>
>