[Make-wifi-fast] emulating wifi better - coupling qdiscs in netem?

Dave Taht dave.taht at gmail.com
Fri Jun 15 18:30:40 EDT 2018

I think tossing netem entirely, ditching the slot models I added to it
last year, and going to userspace to better emulate wifi, is the
answer. Eric just suggested using the iptables NFQUEUE ability to toss
packets to userspace.


nfqueue has batching support built in, so an arbitrary number of
packets can be released as determined by userspace.

# Crappy incorrect pseudocode for setup


# or match on the multicast mac address

iptables -A INPUT -i veth-ap0 -d --j NFQUEUE --queue-num 0

for i in `seq 1 $stations`
iptables -A INPUT -i veth-ap0 -d 10.0.0.$i--j NFQUEUE --queue-num $i

for i in `seq $stations+1 $stations*2`
iptables -A OUTPUT -o veth-ap0 -s 10.0.0.$i --j NFQUEUE --queue-num $i
- ($stations + 1 )

The wifi-emulating daemon then listens on these queues and decides
when to deliver each, and how many packets in a batch.

For wifi, at least, timings are not hugely critical, a few hundred
usec is something userspace can handle reasonably accurately. I like
very much being able to separate out mcast and treat that correctly in
userspace, also. I did want to be below 10usec (wifi "bus"
arbitration), which I am dubious about....

Maybe something "out there" already does this? ns3 comes close... I've
burned the last 4 months of my life trying to do this in-kernel...

Now as for an implementation language? C++ C? Go? Python? The
condition of the wrapper library for go leaves a bit to be desired
( https://github.com/chifflier/nfqueue-go ) and given a choice I'd
MUCH rather use a go than a C.

There is of course a hideous amount of complexity moved to the daemon,
as a pure fifo ap queue forms aggregregates much differently
than a fq_codeled one. But, yea! userspace....

On Wed, May 30, 2018 at 3:57 PM, dpreed at deepplum.com
<dpreed at deepplum.com> wrote:
> I would toss netem rather than kludging around what appears to be a
> fundamental design choice made in ins conceptualization. Make a "netem2".
> FreeBSD has a very nice framework for emulating far more general packet
> queuing/routing/... in the kernel, called NetGraph. It's incredibly general,
> and could straightforwardly, with high performance, have modules that do
> exactly the right emulations of network structures with such blocking, etc.
> and even random delays.
> I know this because in my day job at TidalScale, we heavily use NetGraph to
> implement new very low level protocols, which is pretty straightforward,
> even including complex multi-adapter adaptive forwarding of our private
> protocols on 10 and 40 GigE links. Super flexible, entirely in the kernel,
> running either at real-time priority or not, in a mix.
> In contrast, the Linux TC framework seems very inflexible, as you've found,
> in trying to push it to do what it is not designed to do.
> So tossing netem might be far better. I wonder if NetGraph has ever been
> ported into some Linux kernel environment...
> -----Original Message-----
> From: "Dave Taht" <dave.taht at gmail.com>
> Sent: Wednesday, May 30, 2018 1:28pm
> To: make-wifi-fast at lists.bufferbloat.net
> Subject: [Make-wifi-fast] emulating wifi better - coupling qdiscs in netem?
> The match to reality of my "wifi slotting" code for netem was so
> disappointing that I was extremely reluctant to push support for it up
> to mainline iproute2.
> I've now spent months failing to come up with something that
> could emulate in linux the non-duplex behavior and arbitration steps
> that wifi goes through in order to find a new station to transmit to,
> or receive from, using netem as a base.
> Getting that non-duplex behavior right is the *single most important
> thing*, I think, for trying to emulate real wireless behaviors in
> real time that I can think of (and to thus be able to run and improve
> various e2e transports against it).
> A potential tc API seems simple:
> tc qdisc add dev veth1 root netem coupled # master (AP)
> tc qdisc add dev veth2 root netem couple veth1 # client
> tc qdisc add dev veth3 root netme couple veth2 # client
> Something more complicated would be to create some sort of
> arbitration device and attach that to the qdiscs. (which would make
> it more possible to write arbitration devices to emulate lte, gpon,
> cable, wireless mesh and other non-duplex behaviors in real time)
> But how to convince qdiscs to be arbitrated, only allowing one in a
> set to transmit at the same time? (and worse, in the long run,
> allowing MU-MIMO-like behaviors).
> I'm tempted to *not* put my failed thinking down here in the hope that
> someone says, out there, "oh, that's easy, just create this structure
> with X API call and use Y function and you're clear of all the
> potential deadlock and RCU issues, and we've been doing that for
> years, you idiot! Here's the code for how we do it, sorry we didn't
> submit it earlier."
> What I thought (*and still think*) is of creating a superset of the
> qdisc_watchdog_schedule_ns() function is a start at it:
> tag = qdisc_watchdog_create_arb("some identifier");
> qdisc_watchdog_schedule_arb(nsec, tag); /* null tag = schedule_ns */
> which doesn't allow that qdisc instance to be run until the arbitrator
> says it can run (essentially overriding the timeout specified)
> But I actually wouldn't mind something that worked at the veth, or
> device, rather than qdisc level...
> thoughts?
> PS I just spent several days working on another aspect of the problem,
> which is replaying delay distributions (caused by interference and
> such)... and that, sigh, to me, also belongs in some sort of
> arbitration device rather than directly in netem. Maybe tossing netem
> entirely is the answer. I don't know.
> --
> Dave Täht
> CEO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-669-226-2619
> _______________________________________________
> Make-wifi-fast mailing list
> Make-wifi-fast at lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/make-wifi-fast
> --
> Reed Online Ltd is a company registered in England and Wales. Company
> Registration Number: 6317279.Registered Office: Academy Court, 94 Chancery
> Lane, London WC2A 1DT.


Dave Täht
CEO, TekLibre, LLC
Tel: 1-669-226-2619

More information about the Make-wifi-fast mailing list