From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-qt0-x243.google.com (mail-qt0-x243.google.com [IPv6:2607:f8b0:400d:c0d::243]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id BCBA33B2A4 for ; Fri, 15 Jun 2018 18:30:41 -0400 (EDT) Received: by mail-qt0-x243.google.com with SMTP id l33-v6so10474429qta.11 for ; Fri, 15 Jun 2018 15:30:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=ea72BOCGvOZAIeIh4+VbCQKVF20z99MBxglhuIrCTEQ=; b=lYzSJ17pXQ7e/iWNP028nxM+j5bH0OXzszzbJagmD0WCY8Y390+gGWNxkg7TzRkht3 Qo3xoqTv7twOvzX229u2/bSEUz+LmUbu98QsgHw3CP8zeYB4hOycM8xkng7wQsDKbxrK YtCqdyPWjxDwMFflSYSTnGXfzCm4l8pl2n5wUjGumuXMhHDPNyAH9rhsssrFN2k8dGdh PfXgAhvtSpiseSF72mTAunPml5QNP6IKG4eZPAYZI49JBrby2DrsVWaq27R/H/AUQ0Qs TuNyA3ykdToJlc2p3pXFfyfhj4CW+vxYEvZ/FkCWBqTLd6kNSxNyc8MXsAyOJr69ZaPd xs4Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=ea72BOCGvOZAIeIh4+VbCQKVF20z99MBxglhuIrCTEQ=; b=h8D6vhN1dvQ9dl0BSBAZQPow3Wa7YtEoqDNu+w1pb25Y6aBxsSLER5anZcgBp0UgyC 5NNfU+KCR6bGRjSDBX5zMKLgDaUojo69LD+lTg00WoUAL+f/niOVucPlhwU04/s1BHJm IqKorEvossrzV4lJX5u3k8+Kk2Pl9UaobpquWge1jOySueMDAhPy8JpTrXAJZ7944w2Q PKmPWEL6aKq8R2mgADEiIEdi8NgMXeUewaU3+OnDl5AlD+4tnLsqsh2Km0SQzfqiC9G6 6AG3YBISug1xTKLHT5N0XXDwTgkIIDwDHBF6edmzpnuUJcmLVE1pZDxiX3CfLr8YdYdB RjBw== X-Gm-Message-State: APt69E0oE5tgDajOohnPMiBe3q6i3slAMWYH7WZuKippH1pXdTOxikes 6Eieocc0RNavIdN0lui6NbAsTMp5dHmiwzZahvY= X-Google-Smtp-Source: ADUXVKKz66Bjq7CleeRLjWDxOrGCjb/d3B7bGkowyS5/Txpk9y2mN2iF7oSs6GcBoRg1lagM2GnFx2yf8Erv7HSCQGE= X-Received: by 2002:aed:2311:: with SMTP id h17-v6mr3293372qtc.144.1529101841329; Fri, 15 Jun 2018 15:30:41 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:aed:24f0:0:0:0:0:0 with HTTP; Fri, 15 Jun 2018 15:30:40 -0700 (PDT) In-Reply-To: <1527721073.171416827@apps.rackspace.com> References: <1527721073.171416827@apps.rackspace.com> From: Dave Taht Date: Fri, 15 Jun 2018 15:30:40 -0700 Message-ID: To: "dpreed@deepplum.com" Cc: Make-Wifi-fast Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Subject: Re: [Make-wifi-fast] emulating wifi better - coupling qdiscs in netem? X-BeenThere: make-wifi-fast@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Jun 2018 22:30:41 -0000 I think tossing netem entirely, ditching the slot models I added to it last year, and going to userspace to better emulate wifi, is the answer. Eric just suggested using the iptables NFQUEUE ability to toss packets to userspace. https://home.regit.org/netfilter-en/using-nfqueue-and-libnetfilter_queue/ nfqueue has batching support built in, so an arbitrary number of packets can be released as determined by userspace. # Crappy incorrect pseudocode for setup stations=3D2 # or match on the multicast mac address iptables -A INPUT -i veth-ap0 -d 224.0.0.0/8 --j NFQUEUE --queue-num 0 for i in `seq 1 $stations` do iptables -A INPUT -i veth-ap0 -d 10.0.0.$i--j NFQUEUE --queue-num $i done for i in `seq $stations+1 $stations*2` do iptables -A OUTPUT -o veth-ap0 -s 10.0.0.$i --j NFQUEUE --queue-num $i - ($stations + 1 ) done The wifi-emulating daemon then listens on these queues and decides when to deliver each, and how many packets in a batch. For wifi, at least, timings are not hugely critical, a few hundred usec is something userspace can handle reasonably accurately. I like very much being able to separate out mcast and treat that correctly in userspace, also. I did want to be below 10usec (wifi "bus" arbitration), which I am dubious about.... Maybe something "out there" already does this? ns3 comes close... I've burned the last 4 months of my life trying to do this in-kernel... Now as for an implementation language? C++ C? Go? Python? The condition of the wrapper library for go leaves a bit to be desired ( https://github.com/chifflier/nfqueue-go ) and given a choice I'd MUCH rather use a go than a C. There is of course a hideous amount of complexity moved to the daemon, as a pure fifo ap queue forms aggregregates much differently than a fq_codeled one. But, yea! userspace.... On Wed, May 30, 2018 at 3:57 PM, dpreed@deepplum.com wrote: > I would toss netem rather than kludging around what appears to be a > fundamental design choice made in ins conceptualization. Make a "netem2". > > > > FreeBSD has a very nice framework for emulating far more general packet > queuing/routing/... in the kernel, called NetGraph. It's incredibly gener= al, > and could straightforwardly, with high performance, have modules that do > exactly the right emulations of network structures with such blocking, et= c. > and even random delays. > > > > I know this because in my day job at TidalScale, we heavily use NetGraph = to > implement new very low level protocols, which is pretty straightforward, > even including complex multi-adapter adaptive forwarding of our private > protocols on 10 and 40 GigE links. Super flexible, entirely in the kernel= , > running either at real-time priority or not, in a mix. > > > > In contrast, the Linux TC framework seems very inflexible, as you've foun= d, > in trying to push it to do what it is not designed to do. > > > > So tossing netem might be far better. I wonder if NetGraph has ever been > ported into some Linux kernel environment... > > -----Original Message----- > From: "Dave Taht" > Sent: Wednesday, May 30, 2018 1:28pm > To: make-wifi-fast@lists.bufferbloat.net > Subject: [Make-wifi-fast] emulating wifi better - coupling qdiscs in nete= m? > > The match to reality of my "wifi slotting" code for netem was so > disappointing that I was extremely reluctant to push support for it up > to mainline iproute2. > > I've now spent months failing to come up with something that > could emulate in linux the non-duplex behavior and arbitration steps > that wifi goes through in order to find a new station to transmit to, > or receive from, using netem as a base. > > Getting that non-duplex behavior right is the *single most important > thing*, I think, for trying to emulate real wireless behaviors in > real time that I can think of (and to thus be able to run and improve > various e2e transports against it). > > A potential tc API seems simple: > > tc qdisc add dev veth1 root netem coupled # master (AP) > tc qdisc add dev veth2 root netem couple veth1 # client > tc qdisc add dev veth3 root netme couple veth2 # client > > Something more complicated would be to create some sort of > arbitration device and attach that to the qdiscs. (which would make > it more possible to write arbitration devices to emulate lte, gpon, > cable, wireless mesh and other non-duplex behaviors in real time) > > But how to convince qdiscs to be arbitrated, only allowing one in a > set to transmit at the same time? (and worse, in the long run, > allowing MU-MIMO-like behaviors). > > I'm tempted to *not* put my failed thinking down here in the hope that > someone says, out there, "oh, that's easy, just create this structure > with X API call and use Y function and you're clear of all the > potential deadlock and RCU issues, and we've been doing that for > years, you idiot! Here's the code for how we do it, sorry we didn't > submit it earlier." > > What I thought (*and still think*) is of creating a superset of the > qdisc_watchdog_schedule_ns() function is a start at it: > > tag =3D qdisc_watchdog_create_arb("some identifier"); > qdisc_watchdog_schedule_arb(nsec, tag); /* null tag =3D schedule_ns */ > > which doesn't allow that qdisc instance to be run until the arbitrator > says it can run (essentially overriding the timeout specified) > > But I actually wouldn't mind something that worked at the veth, or > device, rather than qdisc level... > > thoughts? > > PS I just spent several days working on another aspect of the problem, > which is replaying delay distributions (caused by interference and > such)... and that, sigh, to me, also belongs in some sort of > arbitration device rather than directly in netem. Maybe tossing netem > entirely is the answer. I don't know. > > -- > > Dave T=C3=A4ht > CEO, TekLibre, LLC > http://www.teklibre.com > Tel: 1-669-226-2619 > _______________________________________________ > Make-wifi-fast mailing list > Make-wifi-fast@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/make-wifi-fast > -- > Reed Online Ltd is a company registered in England and Wales. Company > Registration Number: 6317279.Registered Office: Academy Court, 94 Chancer= y > Lane, London WC2A 1DT. > --=20 Dave T=C3=A4ht CEO, TekLibre, LLC http://www.teklibre.com Tel: 1-669-226-2619