[Bloat] Re: [Cake] Re: XDP2 is here - from one and only Tom Herbert (almost to the date, 10 years after XDP was released)

["David P. Reed" <dpreed@deepplum.com>]

Tom -
 
An architecture-independent network framework independent of the OS kernel's peculiarities seems within reach (though a fair bit of work), and I think it would be a GOOD THING indeed. IMHO the Linux networking stack in the kernel is a horrific mess, and it doesn't have to be.
 
The reason it doesn't have to be is that there should be no reason it cannot run in ring3/userland, just like DPDK. And it should be built using "real-time" userland programming techniques. (avoiding the generic linux scheduler). The ONLY reason for involving the scheduler would be because there aren't enough cores. Linux was designed to be a uniprocessor Unix, and that just is no longer true at all. With hyperthreading, too, one need never abandon a processor's context in userspace to run some "userland" application.

This would rip a huge amount of kernel code out of the kernel. (at least 50%, and probably more). THe security issues of all those 3rd party network drivers would go away.

And the performance would be much higher for networking.  (running in ring 3, especially if you don't do system calls, is no performance penalty, and interprocessor communications using shared memory is much lower latency than Linux IPC or mutexes).
 
I like the idea of a compilation based network stack, at a slightly higher level than C. eBPF is NOT what I have in mind - it's an interpreter with high overhead. The language should support high-performance co-routining - shared memory, ideally. I don't thing GC is a good thing. Rust might be a good starting point because its memory management is safe.
To me, some of what the base of DPDK is like is good stuff. However, it isn't architecturally neutral.

To me, the network stack should not be entangled with interrupt handling at all. "polling" is far more performant under load. The only use for interrupts is when the network stack is completely idle. That would be, in userland, a "wait for interrupt" call (not a poll). Ideally, on recent Intel machines, a userspace version of MONITOR/MWAIT).

Now I know that Linus and his crew are really NOT gonna like this. Linus is still thinking like MINIX, a uniprocessor time-sharing system with rich OS functions in the kernel and doing "file" reads and writes to communicate with the kernel state. But it is a much more modern way to think of real-time IO in a modern operating system. (Windows and macOS are also Unix-like, uniprocessor monolithic kernel designs).

So, if XDP2 got away from the Linux kernel, it could be great.
BTW, io_uring, etc. are half-measures. They address getting away from interrupts toward polling, but they still make the mistake of keeping huge drivers in the kernel.
 
 
On Tuesday, September 9, 2025 17:36, "Tom Herbert" <tom@herbertland.com> said:









On Tue, Sep 9, 2025, 5:03 PM Frantisek Borsik <[ frantisek.borsik@gmail.com ]( mailto:frantisek.borsik@gmail.com )> wrote:Thanks a lot, David.

 I have asked Tom if he wants to join us and he should be here to chat with
 us now.

 All the best,

 Frank

 Frantisek (Frank) Borsik


 *In loving memory of Dave Täht: *1965-2025

[ https://libreqos.io/2025/04/01/in-loving-memory-of-dave/ ]( https://libreqos.io/2025/04/01/in-loving-memory-of-dave/ )


[ https://www.linkedin.com/in/frantisekborsik ]( https://www.linkedin.com/in/frantisekborsik )

 Signal, Telegram, WhatsApp: +421919416714

 iMessage, mobile: +420775230885

 Skype: casioa5302ca

[ frantisek.borsik@gmail.com ]( mailto:frantisek.borsik@gmail.com )


 On Tue, Sep 9, 2025 at 10:25 PM David P. Reed <[ dpreed@deepplum.com ]( mailto:dpreed@deepplum.com )> wrote:

 > Hi Frank -
 >
 >
 >
 > I think it is interesting as a concept. A project I am advising has been
 > using DPDK very effectively to get rid of the huge path and locking delays
 > in the current Linux network stack. XDP2 could be supported in a ring3
 > (user) address space, achieving a similar result.
HI David,
The idea is you could write the code in XDP2 and it would be compiled to DPDK or eBPF and the compiler would handle the optimizations.
 >
 >
 >
 > But I don't think XDP2 is going that direction - so it may be stuckinto
 > the mess of kernel space networking. Adding eBPF only has made this more of
 > a mess, by the way (and adding a new "compiler" that needs to be veriried
 > as safe for the kernel).
Think of XDP2 as the generalization of XDP to go beyond just the kernel. The idea is that the user writes their datapath code once and they compile it to run in whatever targets they have-- DPDK, P4, other programmable hardware, and yes XDP/eBPF. It's really not limited to kernel networking.
As for the name XDP2, when we created XDP, eXpress DataPath, my vision was that it would be implementation agnostic. eBPF was the first instantiation for practicality, but now ten years later I think we can realize the initial vision.
Tom

>
 > I will be watching how this evolves.
 >
 >
 >
 > David
 >
 >
 >
 > On Tuesday, September 9, 2025 06:32, "Frantisek Borsik" <
 > [ frantisek.borsik@gmail.com ]( mailto:frantisek.borsik@gmail.com )> said:
 >
 > > Hello to all,
 > >
 > > Looks interesting:
 > >
 > [ https://medium.com/@tom_84912/xdp2-this-changes-everything-at-least-for-ai-ml-infrastructure-850c1ba82771 ]( https://medium.com/@tom_84912/xdp2-this-changes-everything-at-least-for-ai-ml-infrastructure-850c1ba82771 )
 > >
 > >
 > > All the best,
 > >
 > > Frank
 > >
 > > Frantisek (Frank) Borsik
 > >
 > >
 > > *In loving memory of Dave Täht: *1965-2025
 > >
 > > [ https://libreqos.io/2025/04/01/in-loving-memory-of-dave/ ]( https://libreqos.io/2025/04/01/in-loving-memory-of-dave/ )
 > >
 > >
 > > [ https://www.linkedin.com/in/frantisekborsik ]( https://www.linkedin.com/in/frantisekborsik )
 > >
 > > Signal, Telegram, WhatsApp: +421919416714
 > >
 > > iMessage, mobile: +420775230885
 > >
 > > Skype: casioa5302ca
 > >
 > > [ frantisek.borsik@gmail.com ]( mailto:frantisek.borsik@gmail.com )
 > > _______________________________________________
 > > Cake mailing list -- [ cake@lists.bufferbloat.net ]( mailto:cake@lists.bufferbloat.net )
 > > To unsubscribe send an email to [ cake-leave@lists.bufferbloat.net ]( mailto:cake-leave@lists.bufferbloat.net )
 > >
 >
 _______________________________________________
 Cake mailing list -- [ cake@lists.bufferbloat.net ]( mailto:cake@lists.bufferbloat.net )
 To unsubscribe send an email to [ cake-leave@lists.bufferbloat.net ]( mailto:cake-leave@lists.bufferbloat.net )