scapy (python) should be able to keep up with a voip or video stream. I've been using it to parse packets and do some other manipulations. It's certainly not C, performance-wise, but it's incredibly flexible at the protocol manipulation level.
The performance issues that I'm running into with it have me tempted to look for something similar in Go. I'm unencapsulating packets (think stripping packets out of a GRE tunnel), and only able to get about 1000-1200 packets/sec on modest hardware (running in a linux VM hosted on windows on an i5 processor...)
-Aaron