[Make-wifi-fast] emulating wifi better - coupling qdiscs in netem?

[Dave Taht]

I think tossing netem entirely, ditching the slot models I added to it
last year, and going to userspace to better emulate wifi, is the
answer. Eric just suggested using the iptables NFQUEUE ability to toss
packets to userspace.

https://home.regit.org/netfilter-en/using-nfqueue-and-libnetfilter_queue/

nfqueue has batching support built in, so an arbitrary number of
packets can be released as determined by userspace.

# Crappy incorrect pseudocode for setup

stations=2

# or match on the multicast mac address

iptables -A INPUT -i veth-ap0 -d 224.0.0.0/8 --j NFQUEUE --queue-num 0

for i in `seq 1 $stations`
do
iptables -A INPUT -i veth-ap0 -d 10.0.0.$i--j NFQUEUE --queue-num $i
done

for i in `seq $stations+1 $stations*2`
do
iptables -A OUTPUT -o veth-ap0 -s 10.0.0.$i --j NFQUEUE --queue-num $i
- ($stations + 1 )
done

The wifi-emulating daemon then listens on these queues and decides
when to deliver each, and how many packets in a batch.

For wifi, at least, timings are not hugely critical, a few hundred
usec is something userspace can handle reasonably accurately. I like
very much being able to separate out mcast and treat that correctly in
userspace, also. I did want to be below 10usec (wifi "bus"
arbitration), which I am dubious about....

Maybe something "out there" already does this? ns3 comes close... I've
burned the last 4 months of my life trying to do this in-kernel...

Now as for an implementation language? C++ C? Go? Python? The
condition of the wrapper library for go leaves a bit to be desired
( https://github.com/chifflier/nfqueue-go ) and given a choice I'd
MUCH rather use a go than a C.

There is of course a hideous amount of complexity moved to the daemon,
as a pure fifo ap queue forms aggregregates much differently
than a fq_codeled one. But, yea! userspace....


On Wed, May 30, 2018 at 3:57 PM, dpreed at deepplum.com
<dpreed at deepplum.com> wrote:
> I would toss netem rather than kludging around what appears to be a
> fundamental design choice made in ins conceptualization. Make a "netem2".
>
>
>
> FreeBSD has a very nice framework for emulating far more general packet
> queuing/routing/... in the kernel, called NetGraph. It's incredibly general,
> and could straightforwardly, with high performance, have modules that do
> exactly the right emulations of network structures with such blocking, etc.
> and even random delays.
>
>
>
> I know this because in my day job at TidalScale, we heavily use NetGraph to
> implement new very low level protocols, which is pretty straightforward,
> even including complex multi-adapter adaptive forwarding of our private
> protocols on 10 and 40 GigE links. Super flexible, entirely in the kernel,
> running either at real-time priority or not, in a mix.
>
>
>
> In contrast, the Linux TC framework seems very inflexible, as you've found,
> in trying to push it to do what it is not designed to do.
>
>
>
> So tossing netem might be far better. I wonder if NetGraph has ever been
> ported into some Linux kernel environment...
>
> -----Original Message-----
> From: "Dave Taht" <dave.taht at gmail.com>
> Sent: Wednesday, May 30, 2018 1:28pm
> To: make-wifi-fast at lists.bufferbloat.net
> Subject: [Make-wifi-fast] emulating wifi better - coupling qdiscs in netem?
>
> The match to reality of my "wifi slotting" code for netem was so
> disappointing that I was extremely reluctant to push support for it up
> to mainline iproute2.
>
> I've now spent months failing to come up with something that
> could emulate in linux the non-duplex behavior and arbitration steps
> that wifi goes through in order to find a new station to transmit to,
> or receive from, using netem as a base.
>
> Getting that non-duplex behavior right is the *single most important
> thing*, I think, for trying to emulate real wireless behaviors in
> real time that I can think of (and to thus be able to run and improve
> various e2e transports against it).
>
> A potential tc API seems simple:
>
> tc qdisc add dev veth1 root netem coupled # master (AP)
> tc qdisc add dev veth2 root netem couple veth1 # client
> tc qdisc add dev veth3 root netme couple veth2 # client
>
> Something more complicated would be to create some sort of
> arbitration device and attach that to the qdiscs. (which would make
> it more possible to write arbitration devices to emulate lte, gpon,
> cable, wireless mesh and other non-duplex behaviors in real time)
>
> But how to convince qdiscs to be arbitrated, only allowing one in a
> set to transmit at the same time? (and worse, in the long run,
> allowing MU-MIMO-like behaviors).
>
> I'm tempted to *not* put my failed thinking down here in the hope that
> someone says, out there, "oh, that's easy, just create this structure
> with X API call and use Y function and you're clear of all the
> potential deadlock and RCU issues, and we've been doing that for
> years, you idiot! Here's the code for how we do it, sorry we didn't
> submit it earlier."
>
> What I thought (*and still think*) is of creating a superset of the
> qdisc_watchdog_schedule_ns() function is a start at it:
>
> tag = qdisc_watchdog_create_arb("some identifier");
> qdisc_watchdog_schedule_arb(nsec, tag); /* null tag = schedule_ns */
>
> which doesn't allow that qdisc instance to be run until the arbitrator
> says it can run (essentially overriding the timeout specified)
>
> But I actually wouldn't mind something that worked at the veth, or
> device, rather than qdisc level...
>
> thoughts?
>
> PS I just spent several days working on another aspect of the problem,
> which is replaying delay distributions (caused by interference and
> such)... and that, sigh, to me, also belongs in some sort of
> arbitration device rather than directly in netem. Maybe tossing netem
> entirely is the answer. I don't know.
>
> --
>
> Dave Täht
> CEO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-669-226-2619
> _______________________________________________
> Make-wifi-fast mailing list
> Make-wifi-fast at lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/make-wifi-fast
> --
> Reed Online Ltd is a company registered in England and Wales. Company
> Registration Number: 6317279.Registered Office: Academy Court, 94 Chancery
> Lane, London WC2A 1DT.
>



-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619