From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wr0-x242.google.com (mail-wr0-x242.google.com [IPv6:2a00:1450:400c:c0c::242]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id CFCD93BA8E for ; Sun, 17 Jun 2018 11:16:34 -0400 (EDT) Received: by mail-wr0-x242.google.com with SMTP id l41-v6so14251003wre.7 for ; Sun, 17 Jun 2018 08:16:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=heistp.net; s=google; h=mime-version:subject:from:in-reply-to:date:cc:message-id:references :to; bh=ULWYsBk0dahjIaF+HbaSOwBmG+iU7o6MmzLbvq9zxXE=; b=Eby5I766sHlJGkA34pp3CoIGbOawlMovyxt0YFtwtSHJfiA5cUhLpEyqV4e5WikrM6 TIpMy36COgYzRNof/M2Ek2h7gnOFnUOPbqgX0uGt05UuKgxYcL1XcCiSqngMhDwvX5s9 3zNsn5rlWo6jy1JEYj0nFzOlFBhoXX95TJdH7DHUfYqa+q2Fz2xO5btRFdZpxaRGYqG0 /v0r8A/WmqF3lwedPPXMVHiV9+tyx0VRyQg66uLRu2YQwyl5HoXehfSt+bK3OYoWrBMX uC2ZkfXFJ83m0hIFg+kvNSvI4F/0XGPzwT3JDPR+zzyLarcBtAsNng3/a1iEv2Knq1HC Fg7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :message-id:references:to; bh=ULWYsBk0dahjIaF+HbaSOwBmG+iU7o6MmzLbvq9zxXE=; b=gT6K8PNPXVlmesYKB9yUDW0i6gTjD1UUawQ0Ps95mVhKKThCSDqb3Z+jRMk080Vvpj IaPdE37v7odUSz4GMK6/ivnkYoUPYhqU2ZLQZXGt2GOxQFkoddhAyATkhGrzhOJoTqBQ ckoHk2RCFF4Hf04T/gItyBVAFYDz6y5aDTuCxVvMSr3ipqHBuvXv5/3c0bfpLeFsckyi UxlhVQbSeytYgYj49/y8jxxAzne2hH9EJoAmulNtKBkJff/ztFghD3J/wC80XggwYJNU 141TkbMz3zJkJI+gyUmeba1UGTMH2QDcxOaWuSfgcmrwbC0rVHnduR2eeBoiQGxJWe/c dFwQ== X-Gm-Message-State: APt69E0F5ARiLGR+Jgl+CtyRzZ4EVtpZYGZoa08seZkR1wl5U6xFFcCu v4MZG/B4gSi6FRtvDsCbLIgfTA== X-Google-Smtp-Source: ADUXVKI3vkSAqO9ts/Yte1uX8p2QXt2oRy1+Szxq04GeGDI2S15B9HyWpn1Ky1nf95GbzVBeIRNRTA== X-Received: by 2002:adf:9724:: with SMTP id r33-v6mr7977794wrb.79.1529248593496; Sun, 17 Jun 2018 08:16:33 -0700 (PDT) Received: from tron.luk.heistp.net (h-1169.lbcfree.net. [185.193.85.130]) by smtp.gmail.com with ESMTPSA id n18-v6sm15259198wrj.58.2018.06.17.08.16.32 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sun, 17 Jun 2018 08:16:33 -0700 (PDT) Content-Type: multipart/alternative; boundary="Apple-Mail=_AEE169A4-5937-45CC-9B80-F50868A37B70" Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) From: Pete Heist In-Reply-To: <20180617131921.09bf5353@redhat.com> Date: Sun, 17 Jun 2018 17:16:30 +0200 Cc: Dave Taht , Florian Westphal , Marek Majkowski , Make-Wifi-fast Message-Id: References: <1527721073.171416827@apps.rackspace.com> <150ABF21-FAFC-48E2-9E55-CAA609EAE449@heistp.net> <20180617131921.09bf5353@redhat.com> To: Jesper Dangaard Brouer X-Mailer: Apple Mail (2.3124) Subject: Re: [Make-wifi-fast] emulating wifi better - coupling qdiscs in netem? X-BeenThere: make-wifi-fast@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 17 Jun 2018 15:16:35 -0000 --Apple-Mail=_AEE169A4-5937-45CC-9B80-F50868A37B70 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Hi Jesper/Florian, thanks for noticing that, not surprisingly it = doesn=E2=80=99t change the ping results much, but it improves throughput = a lot (now only ~20% less than without nfqueue): root@lsrv:~# iperf3 -t 5 -c 10.182.122.11 Connecting to host 10.182.122.11, port 5201 [ 4] local 10.182.122.1 port 55936 connected to 10.182.122.11 port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 375 MBytes 3.14 Gbits/sec 173 372 KBytes = =20 [ 4] 1.00-2.00 sec 365 MBytes 3.06 Gbits/sec 316 382 KBytes = =20 [ 4] 2.00-3.00 sec 372 MBytes 3.13 Gbits/sec 368 427 KBytes = =20 [ 4] 3.00-4.00 sec 364 MBytes 3.05 Gbits/sec 137 402 KBytes = =20 [ 4] 4.00-5.00 sec 364 MBytes 3.05 Gbits/sec 342 382 KBytes = =20 - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-5.00 sec 1.80 GBytes 3.09 Gbits/sec 1336 = sender [ 4] 0.00-5.00 sec 1.79 GBytes 3.08 Gbits/sec = receiver iperf Done. I don=E2=80=99t know if/how the use of GSO affects Dave=E2=80=99s = simulation work, but I=E2=80=99ll leave that to him. I only wanted to = contribute a quick evaluation. :) Pete > On Jun 17, 2018, at 1:19 PM, Jesper Dangaard Brouer = wrote: >=20 >=20 > Hi Pete, >=20 > Happened to be at the Netfilter Workshop, and discussed nfqueue with > Florian and Marek, and I saw this attempt to use nfqueue, and Florian > points out that you are not using the GRO facility of nfqueue. >=20 > I'll quote what Florian said below: >=20 > On Sun, 17 Jun 2018 12:45:52 +0200 Florian Westphal > wrote: >=20 >> The linked example code is old and does not set >> mnl_attr_put_u32(nlh, NFQA_CFG_FLAGS, htonl(NFQA_CFG_F_GSO)); >>=20 >> When requesting the queue. >>=20 >> This means kernel has to do software segmentation of GSO skbs. >>=20 >> Consider using >> https://git.netfilter.org/libnetfilter_queue/tree/examples/nf-queue.c = >>=20 >> instead if you need a template, it does this correctly. >=20 > --Jesper >=20 >=20 > On Sun, 17 Jun 2018 00:53:03 +0200 Pete Heist > wrote: >=20 >>> On Jun 16, 2018, at 12:30 AM, Dave Taht > wrote: >>>=20 >>> Eric just suggested using the iptables NFQUEUE ability to toss >>> packets to userspace. >>>=20 >>> = https://home.regit.org/netfilter-en/using-nfqueue-and-libnetfilter_queue/ = = > >>> For wifi, at least, timings are not hugely critical, a few hundred >>> usec is something userspace can handle reasonably accurately. I like >>> very much being able to separate out mcast and treat that correctly = in >>> userspace, also. I did want to be below 10usec (wifi "bus" >>> arbitration), which I am dubious about.... >>>=20 >>> Now as for an implementation language? C++ C? Go? Python? The >>> condition of the wrapper library for go leaves a bit to be desired >>> ( https://github.com/chifflier/nfqueue-go = = > ) and given a choice I'd >>> MUCH rather use a go than a C. =20 >>=20 >> This sounds cool... So for fun, I compared ping and iperf3 with no-op = nfqueue callbacks in both C and Go. As for the hardware setup, I used = two lxc containers (effectively just veth) on an APU2. >>=20 >> For the Go program, I used test_nfqueue from the wrapper above (which = yes, does need some work) and removed debugging / logging. >>=20 >> For the C program I used this: >> = https://github.com/irontec/netfilter-nfqueue-samples/blob/master/sample-he= lloworld.c >> I removed any per-packet printf calls and compiled with "gcc = sample-helloworld.c -o nfq -lnfnetlink -lnetfilter_queue=E2=80=9D. >>=20 >> Ping results: >>=20 >> ping without nfqueue: >> root@lsrv:~# iptables -F OUTPUT >> root@lsrv:~# ping -c 500 -i 0.01 -q 10.182.122.11 >> 500 packets transmitted, 500 received, 0% packet loss, time 7985ms >> rtt min/avg/max/mdev =3D 0.056/0.058/0.185/0.011 ms >>=20 >> ping with no-op nfqueue callback in C: >> root@lsrv:~# iptables -A OUTPUT -d 10.182.122.11/32 -j NFQUEUE = --queue-num 0 >> root@lsrv:~/nfqueue# ping -c 500 -i 0.01 -q 10.182.122.11 >> 500 packets transmitted, 500 received, 0% packet loss, time 7981ms >> rtt min/avg/max/mdev =3D 0.117/0.123/0.384/0.020 ms >>=20 >> ping with no-op nfqueue callback in Go: >> root@lsrv:~# iptables -A OUTPUT -d 10.182.122.11/32 -j NFQUEUE = --queue-num 0 >> root@lsrv:~# ping -c 500 -i 0.01 -q 10.182.122.11 >> 500 packets transmitted, 500 received, 0% packet loss, time 7982ms >> rtt min/avg/max/mdev =3D 0.095/0.172/0.532/0.042 ms >>=20 >> The mean induced latency of 65us for C or 114us for Go might be = within your parameters, except you mentioned 10us for WiFi bus = arbitration, which does indeed look impossible with this setup, even in = C. >>=20 >> Iperf3 results: >>=20 >> iperf3 without nfqueue: >> root@lsrv:~# iptables -F OUTPUT >> root@lsrv:~# iperf3 -t 5 -c 10.182.122.11 >> Connecting to host 10.182.122.11, port 5201 >> [ 4] local 10.182.122.1 port 55810 connected to 10.182.122.11 port = 5201 >> [ ID] Interval Transfer Bandwidth Retr Cwnd >> [ 4] 0.00-1.00 sec 452 MBytes 3.79 Gbits/sec 0 178 = KBytes =20 >> [ 4] 1.00-2.00 sec 454 MBytes 3.82 Gbits/sec 0 320 = KBytes =20 >> [ 4] 2.00-3.00 sec 450 MBytes 3.77 Gbits/sec 0 320 = KBytes =20 >> [ 4] 3.00-4.00 sec 451 MBytes 3.79 Gbits/sec 0 352 = KBytes =20 >> [ 4] 4.00-5.00 sec 451 MBytes 3.79 Gbits/sec 0 352 = KBytes =20 >> - - - - - - - - - - - - - - - - - - - - - - - - - >> [ ID] Interval Transfer Bandwidth Retr >> [ 4] 0.00-5.00 sec 2.21 GBytes 3.79 Gbits/sec 0 = sender >> [ 4] 0.00-5.00 sec 2.21 GBytes 3.79 Gbits/sec = receiver >> iperf Done. >>=20 >> iperf3 with no-op nfqueue callback in C: >> root@lsrv:~# iptables -A OUTPUT -d 10.182.122.11/32 -j NFQUEUE = --queue-num 0 >> root@lsrv:~/nfqueue# iperf3 -t 5 -c 10.182.122.11 >> Connecting to host 10.182.122.11, port 5201 >> [ 4] local 10.182.122.1 port 55868 connected to 10.182.122.11 port = 5201 >> [ ID] Interval Transfer Bandwidth Retr Cwnd >> [ 4] 0.00-1.00 sec 17.4 MBytes 146 Mbits/sec 0 107 = KBytes =20 >> [ 4] 1.00-2.00 sec 16.9 MBytes 142 Mbits/sec 0 107 = KBytes =20 >> [ 4] 2.00-3.00 sec 17.0 MBytes 142 Mbits/sec 0 107 = KBytes =20 >> [ 4] 3.00-4.00 sec 17.0 MBytes 142 Mbits/sec 0 107 = KBytes =20 >> [ 4] 4.00-5.00 sec 17.0 MBytes 143 Mbits/sec 0 115 = KBytes =20 >> - - - - - - - - - - - - - - - - - - - - - - - - - >> [ ID] Interval Transfer Bandwidth Retr >> [ 4] 0.00-5.00 sec 85.3 MBytes 143 Mbits/sec 0 = sender >> [ 4] 0.00-5.00 sec 84.7 MBytes 142 Mbits/sec = receiver >>=20 >> iperf3 with no-op nfqueue callback in Go: >> root@lsrv:~# iptables -A OUTPUT -d 10.182.122.11/32 -j NFQUEUE = --queue-num 0 >> root@lsrv:~# iperf3 -t 5 -c 10.182.122.11 >> Connecting to host 10.182.122.11, port 5201 >> [ 4] local 10.182.122.1 port 55864 connected to 10.182.122.11 port = 5201 >> [ ID] Interval Transfer Bandwidth Retr Cwnd >> [ 4] 0.00-1.00 sec 14.6 MBytes 122 Mbits/sec 0 96.2 = KBytes =20 >> [ 4] 1.00-2.00 sec 14.1 MBytes 118 Mbits/sec 0 96.2 = KBytes =20 >> [ 4] 2.00-3.00 sec 14.0 MBytes 118 Mbits/sec 0 102 = KBytes =20 >> [ 4] 3.00-4.00 sec 14.0 MBytes 117 Mbits/sec 0 102 = KBytes =20 >> [ 4] 4.00-5.00 sec 13.7 MBytes 115 Mbits/sec 0 107 = KBytes =20 >> - - - - - - - - - - - - - - - - - - - - - - - - - >> [ ID] Interval Transfer Bandwidth Retr >> [ 4] 0.00-5.00 sec 70.5 MBytes 118 Mbits/sec 0 = sender >> [ 4] 0.00-5.00 sec 69.9 MBytes 117 Mbits/sec = receiver >> iperf Done. >>=20 >> So rats, throughput gets brutalized for both C and Go. For Go, a rate = of 117 Mbit with a 1500 byte MTU is 9750 packets/sec, which is 103us / = packet. Mean induced latency measured by ping is 114us, which is not far = off 103us, so the rate slowdown looks to be mostly caused by the = per-packet nfqueue calls. The core running test_nfqueue is pinned at = 100% during the test. "nice -n -20" does nothing. >>=20 >> Presumably you=E2=80=99ll sometimes be releasing more than one packet = at a time(?) so I guess whether or not this is workable depends on how = many you release at once, what hardware you=E2=80=99re on and what rates = you need to test at. But when you=E2=80=99re trying to test a qdisc, I = guess you=E2=80=99d want to minimize the burden you add to the CPU, or = else move it to a core the qdisc isn=E2=80=99t running on, or something, = so the qdisc itself isn=E2=80=99t affected by the test rig. >>=20 >>> There is of course a hideous amount of complexity moved to the = daemon, =20 >>=20 >> I can only imagine. >>=20 >>> as a pure fifo ap queue forms aggregregates much differently >>> than a fq_codeled one. But, yea! userspace.... =20 >>=20 >> This would be awesome if it works out! After that iperf3 test though, = I think I may have smashed my dreams of writing a libnetfilter_queue = userspace qdisc in Go, or C for that matter. >>=20 >> If this does somehow turn out to be good enough performance-wise, I = think you=E2=80=99d have a lot more fun and spend a lot less time on it = in Go than C, but that=E2=80=99s just an opinion... :) >>=20 >=20 >=20 >=20 > --=20 > Best regards, > Jesper Dangaard Brouer > MSc.CS, Principal Kernel Engineer at Red Hat > LinkedIn: http://www.linkedin.com/in/brouer = --Apple-Mail=_AEE169A4-5937-45CC-9B80-F50868A37B70 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8
Hi Jesper/Florian, thanks for noticing that, = not surprisingly it doesn=E2=80=99t change the ping results much, but it = improves throughput a lot (now only ~20% less than without = nfqueue):

root@lsrv:~# iperf3 -t 5 -c 10.182.122.11
Connecting to host 10.182.122.11, port 5201
[ =  4] local 10.182.122.1 port 55936 connected to 10.182.122.11 port = 5201
[ ID] Interval           = Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec   375 MBytes =  3.14 Gbits/sec  173    372 KBytes    =    
[  4]   1.00-2.00   sec =   365 MBytes  3.06 Gbits/sec  316    382 = KBytes       
[  4]   = 2.00-3.00   sec   372 MBytes  3.13 Gbits/sec  368 =    427 KBytes       
[ =  4]   3.00-4.00   sec   364 MBytes  3.05 = Gbits/sec  137    402 KBytes      =  
[  4]   4.00-5.00   sec   364 = MBytes  3.05 Gbits/sec  342    382 KBytes  =      
- - - - - - - - - - - - - - - - - - - = - - - - - -
[ ID] Interval         =   Transfer     Bandwidth       Retr
[  4]   0.00-5.00   sec  1.80 GBytes =  3.09 Gbits/sec  1336           =   sender
[  4]   0.00-5.00   sec =  1.79 GBytes  3.08 Gbits/sec         =          receiver
iperf = Done.

I = don=E2=80=99t know if/how the use of GSO affects Dave=E2=80=99s = simulation work, but I=E2=80=99ll leave that to him. I only wanted to = contribute a quick evaluation. :)

Pete

On Jun 17, 2018, at 1:19 PM, Jesper Dangaard = Brouer <brouer@redhat.com> wrote:


Hi Pete,

Happened to be at the Netfilter Workshop, = and discussed nfqueue with
Florian and Marek, and I saw this attempt = to use nfqueue, and Florian
points out that you are not using the GRO = facility of nfqueue.

I'll quote what Florian said below:

On Sun, 17 Jun 2018 12:45:52 +0200 = Florian Westphal <fw@strlen.de> wrote:

The = linked example code is old and does not set
= mnl_attr_put_u32(nlh, NFQA_CFG_FLAGS, htonl(NFQA_CFG_F_GSO));

When requesting the queue.

This means kernel has to do software segmentation of GSO = skbs.

Consider using
https://git.netfilter.org/libnetfilter_queue/tree/examples/nf-q= ueue.c

instead if you need a template, = it does this correctly.

--Jesper


On Sun, 17 Jun 2018 00:53:03 +0200 Pete Heist = <pete@heistp.net> = wrote:

On Jun 16, 2018, at 12:30 AM, Dave Taht <dave.taht@gmail.com>= wrote:

Eric just suggested using the = iptables NFQUEUE ability to toss
packets to userspace.

https://home.regit.org/netfilter-en/using-nfqueue-and-libnetfil= ter_queue/ <https://home.regit.org/netfilter-en/using-nfqueue-and-libnetfil= ter_queue/>
For wifi, at least, timings are not = hugely critical, a few hundred
usec is something userspace = can handle reasonably accurately. I like
very much being = able to separate out mcast and treat that correctly in
userspace, also. I did want to be below 10usec (wifi "bus"
arbitration), which I am dubious about....

Now as for an implementation language? C++ C? Go? Python? = The
condition of the wrapper library for go leaves a bit = to be desired
( https://github.com/chifflier/nfqueue-go <https://github.com/chifflier/nfqueue-go> ) and given a = choice I'd
MUCH rather use a go than a C.  

This sounds cool... So for fun, I = compared ping and iperf3 with no-op nfqueue callbacks in both C and Go. = As for the hardware setup, I used two lxc containers (effectively just = veth) on an APU2.

For the Go program, I = used test_nfqueue from the wrapper above (which yes, does need some = work) and removed debugging / logging.

For = the C program I used this:
https://github.com/irontec/netfilter-nfqueue-samples/blob/maste= r/sample-helloworld.c
I removed any per-packet printf = calls and compiled with "gcc sample-helloworld.c -o nfq -lnfnetlink = -lnetfilter_queue=E2=80=9D.

Ping = results:

ping without nfqueue:
root@lsrv:~# iptables -F OUTPUT
root@lsrv:~# = ping -c 500 -i 0.01 -q 10.182.122.11
500 packets = transmitted, 500 received, 0% packet loss, time 7985ms
rtt = min/avg/max/mdev =3D 0.056/0.058/0.185/0.011 ms

ping with no-op nfqueue callback in C:
root@lsrv:~# iptables -A OUTPUT -d 10.182.122.11/32 -j = NFQUEUE --queue-num 0
root@lsrv:~/nfqueue# ping -c 500 -i = 0.01 -q 10.182.122.11
500 packets transmitted, 500 = received, 0% packet loss, time 7981ms
rtt min/avg/max/mdev = =3D 0.117/0.123/0.384/0.020 ms

ping with = no-op nfqueue callback in Go:
root@lsrv:~# iptables -A = OUTPUT -d 10.182.122.11/32 -j NFQUEUE --queue-num 0
root@lsrv:~# ping -c 500 -i 0.01 -q 10.182.122.11
500 packets transmitted, 500 received, 0% packet loss, time = 7982ms
rtt min/avg/max/mdev =3D 0.095/0.172/0.532/0.042 = ms

The mean induced latency of 65us for C = or 114us for Go might be within your parameters, except you mentioned = 10us for WiFi bus arbitration, which does indeed look impossible with = this setup, even in C.

Iperf3 results:

iperf3 without nfqueue:
root@lsrv:~# iptables -F OUTPUT
root@lsrv:~# = iperf3 -t 5 -c 10.182.122.11
Connecting to host = 10.182.122.11, port 5201
[  4] local 10.182.122.1 = port 55810 connected to 10.182.122.11 port 5201
[ ID] = Interval =           Transfer =     Bandwidth =       Retr  Cwnd
[ =  4]   0.00-1.00   sec   452 MBytes =  3.79 Gbits/sec    0    178 KBytes =       
[  4] =   1.00-2.00   sec   454 MBytes  3.82 = Gbits/sec    0    320 KBytes =       
[  4] =   2.00-3.00   sec   450 MBytes  3.77 = Gbits/sec    0    320 KBytes =       
[  4] =   3.00-4.00   sec   451 MBytes  3.79 = Gbits/sec    0    352 KBytes =       
[  4] =   4.00-5.00   sec   451 MBytes  3.79 = Gbits/sec    0    352 KBytes =       
- - - - - - - - - - - = - - - - - - - - - - - - - -
[ ID] Interval =           Transfer =     Bandwidth =       Retr
[  4] =   0.00-5.00   sec  2.21 GBytes  3.79 = Gbits/sec    0 =             se= nder
[  4]   0.00-5.00   sec =  2.21 GBytes  3.79 Gbits/sec =             &n= bsp;    receiver
iperf Done.

iperf3 with no-op nfqueue callback in C:
root@lsrv:~# iptables -A OUTPUT -d 10.182.122.11/32 -j = NFQUEUE --queue-num 0
root@lsrv:~/nfqueue# iperf3 -t 5 -c = 10.182.122.11
Connecting to host 10.182.122.11, port = 5201
[  4] local 10.182.122.1 port 55868 connected to = 10.182.122.11 port 5201
[ ID] Interval =           Transfer =     Bandwidth =       Retr  Cwnd
[ =  4]   0.00-1.00   sec  17.4 MBytes =   146 Mbits/sec    0    107 = KBytes       
[  4] =   1.00-2.00   sec  16.9 MBytes   142 = Mbits/sec    0    107 KBytes =       
[  4] =   2.00-3.00   sec  17.0 MBytes   142 = Mbits/sec    0    107 KBytes =       
[  4] =   3.00-4.00   sec  17.0 MBytes   142 = Mbits/sec    0    107 KBytes =       
[  4] =   4.00-5.00   sec  17.0 MBytes   143 = Mbits/sec    0    115 KBytes =       
- - - - - - - - - - - = - - - - - - - - - - - - - -
[ ID] Interval =           Transfer =     Bandwidth =       Retr
[  4] =   0.00-5.00   sec  85.3 MBytes   143 = Mbits/sec    0 =             se= nder
[  4]   0.00-5.00   sec =  84.7 MBytes   142 Mbits/sec =             &n= bsp;    receiver

iperf3 = with no-op nfqueue callback in Go:
root@lsrv:~# iptables = -A OUTPUT -d 10.182.122.11/32 -j NFQUEUE --queue-num 0
root@lsrv:~# iperf3 -t 5 -c 10.182.122.11
Connecting to host 10.182.122.11, port 5201
[ =  4] local 10.182.122.1 port 55864 connected to 10.182.122.11 port = 5201
[ ID] Interval =           Transfer =     Bandwidth =       Retr  Cwnd
[ =  4]   0.00-1.00   sec  14.6 MBytes =   122 Mbits/sec    0   96.2 KBytes =       
[  4] =   1.00-2.00   sec  14.1 MBytes   118 = Mbits/sec    0   96.2 KBytes =       
[  4] =   2.00-3.00   sec  14.0 MBytes   118 = Mbits/sec    0    102 KBytes =       
[  4] =   3.00-4.00   sec  14.0 MBytes   117 = Mbits/sec    0    102 KBytes =       
[  4] =   4.00-5.00   sec  13.7 MBytes   115 = Mbits/sec    0    107 KBytes =       
- - - - - - - - - - - = - - - - - - - - - - - - - -
[ ID] Interval =           Transfer =     Bandwidth =       Retr
[  4] =   0.00-5.00   sec  70.5 MBytes   118 = Mbits/sec    0 =             se= nder
[  4]   0.00-5.00   sec =  69.9 MBytes   117 Mbits/sec =             &n= bsp;    receiver
iperf Done.

So rats, throughput gets brutalized for both C = and Go. For Go, a rate of 117 Mbit with a 1500 byte MTU is 9750 = packets/sec, which is 103us / packet. Mean induced latency measured by = ping is 114us, which is not far off 103us, so the rate slowdown looks to = be mostly caused by the per-packet nfqueue calls. The core running = test_nfqueue is pinned at 100% during the test. "nice -n -20" does = nothing.

Presumably you=E2=80=99ll = sometimes be releasing more than one packet at a time(?) so I guess = whether or not this is workable depends on how many you release at once, = what hardware you=E2=80=99re on and what rates you need to test at. But = when you=E2=80=99re trying to test a qdisc, I guess you=E2=80=99d want = to minimize the burden you add to the CPU, or else move it to a core the = qdisc isn=E2=80=99t running on, or something, so the qdisc itself = isn=E2=80=99t affected by the test rig.

There is of course a = hideous amount of complexity moved to the daemon,  

I can only imagine.

as a pure = fifo ap queue forms aggregregates much differently
than a = fq_codeled one. But, yea! userspace....  

This would be awesome if it works = out! After that iperf3 test though, I think I may have smashed my dreams = of writing a libnetfilter_queue userspace qdisc in Go, or C for that = matter.

If this does somehow turn out to be = good enough performance-wise, I think you=E2=80=99d have a lot more fun = and spend a lot less time on it in Go than C, but that=E2=80=99s just an = opinion... :)




-- 
Best = regards,
 Jesper Dangaard Brouer
 MSc.CS, = Principal Kernel Engineer at Red Hat
 LinkedIn: http://www.linkedin.com/in/brouer
<= br class=3D"">= --Apple-Mail=_AEE169A4-5937-45CC-9B80-F50868A37B70--