From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wr0-x243.google.com (mail-wr0-x243.google.com [IPv6:2a00:1450:400c:c0c::243]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id AE9A03B29E for ; Sat, 16 Jun 2018 18:53:05 -0400 (EDT) Received: by mail-wr0-x243.google.com with SMTP id l41-v6so13153124wre.7 for ; Sat, 16 Jun 2018 15:53:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=heistp.net; s=google; h=mime-version:subject:from:in-reply-to:date:cc:message-id:references :to; bh=ou013H7am7WEsuAS8k7Ko5WvcF0xPl0AQATTLPn1P0w=; b=PJ7KRT/ASONpPatiOqQRyOYh5BwmWeIBC6TjA6s7wYM0RzYDNxn5IPmhc87AMJ3NeF lAkBjbgpZVksZwM0BoxlAOkqU7r/ZxPsOK7YlQH3PGUKf8SDjCKt6QZfAr/YQzTarYSH N8A0RfxwZ7ofsDcNxtv5YdK75fCw+6fSJhoOvC3I930da8ON1det7HFMm5hoZsm+0QL+ lq4dFJC84q+VX9FbV+Tal1UATx4Z0CiFaWfTv4r8AwuvP0riPjz+q+0SVbb5k+SAG/sT F+8tHCIvhkjVDBkO2cK0UttqRka8akc2H6UMmc9UzxfQdl5uiAl8c7qq3bGfWxKNaqx6 mJ9A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :message-id:references:to; bh=ou013H7am7WEsuAS8k7Ko5WvcF0xPl0AQATTLPn1P0w=; b=H8KmjhWJPU1kxKrQ0oqoTMzcK85jlHItZ5anSkoTxglOZFYjH/1LL1ClabKEuD8nM6 oHVhn/fV5DpqnTgfOp05Tkpo3d+kXPZgYc3dfFZNjei9vEWrlD8b9aN5oEyBn2FVdXO+ nvPZLgN5ENl5f0tekTtX3lBrUzIjb11O3vj/qxY51PfOECrhKb9mtVlaBKkeDl6jcDFd RG772DC9IufJiyvKnko2sHyCx49+AXRIoIr9MjIPpanVWoJQkUsCpXYTIgRr1p9ZIfiO V3FzjNxjOSZ2nQmrw/tAV3ZdNdoaVIGAULIdGCFxLS5pVrfcq5j2HF+RjSSyEeGv15xu LSfA== X-Gm-Message-State: APt69E2Y/PPKqD5VXAC2k/CzA72NKIVx9/ZXOaH99Lu3FIm4AMjaW7p/ TPmBmPbqAEnCbvgvpCfR+1Nza17Kvyc= X-Google-Smtp-Source: ADUXVKK+Fo0cC+U8IWf/YeqkrOutUjjfdRv8fAlHlszZcdEZK+MAIN6ViMD5/YF29YRGf/ypq2lLEg== X-Received: by 2002:adf:82d1:: with SMTP id 75-v6mr5636430wrc.77.1529189584725; Sat, 16 Jun 2018 15:53:04 -0700 (PDT) Received: from tron.luk.heistp.net (h-1169.lbcfree.net. [185.193.85.130]) by smtp.gmail.com with ESMTPSA id n17-v6sm7230535wmd.14.2018.06.16.15.53.03 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sat, 16 Jun 2018 15:53:04 -0700 (PDT) Content-Type: multipart/alternative; boundary="Apple-Mail=_D879296B-C415-4DB5-B5D9-A9D31D0F199B" Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) From: Pete Heist In-Reply-To: Date: Sun, 17 Jun 2018 00:53:03 +0200 Cc: Make-Wifi-fast Message-Id: <150ABF21-FAFC-48E2-9E55-CAA609EAE449@heistp.net> References: <1527721073.171416827@apps.rackspace.com> To: Dave Taht X-Mailer: Apple Mail (2.3124) Subject: Re: [Make-wifi-fast] emulating wifi better - coupling qdiscs in netem? X-BeenThere: make-wifi-fast@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 16 Jun 2018 22:53:06 -0000 --Apple-Mail=_D879296B-C415-4DB5-B5D9-A9D31D0F199B Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On Jun 16, 2018, at 12:30 AM, Dave Taht wrote: >=20 > Eric just suggested using the iptables NFQUEUE ability to toss > packets to userspace. >=20 > = https://home.regit.org/netfilter-en/using-nfqueue-and-libnetfilter_queue/ = > For wifi, at least, timings are not hugely critical, a few hundred > usec is something userspace can handle reasonably accurately. I like > very much being able to separate out mcast and treat that correctly in > userspace, also. I did want to be below 10usec (wifi "bus" > arbitration), which I am dubious about.... >=20 > Now as for an implementation language? C++ C? Go? Python? The > condition of the wrapper library for go leaves a bit to be desired > ( https://github.com/chifflier/nfqueue-go = ) and given a choice I'd > MUCH rather use a go than a C. This sounds cool... So for fun, I compared ping and iperf3 with no-op = nfqueue callbacks in both C and Go. As for the hardware setup, I used = two lxc containers (effectively just veth) on an APU2. For the Go program, I used test_nfqueue from the wrapper above (which = yes, does need some work) and removed debugging / logging. For the C program I used this: = https://github.com/irontec/netfilter-nfqueue-samples/blob/master/sample-he= lloworld.c I removed any per-packet printf calls and compiled with "gcc = sample-helloworld.c -o nfq -lnfnetlink -lnetfilter_queue=E2=80=9D. Ping results: ping without nfqueue: root@lsrv:~# iptables -F OUTPUT root@lsrv:~# ping -c 500 -i 0.01 -q 10.182.122.11 500 packets transmitted, 500 received, 0% packet loss, time 7985ms rtt min/avg/max/mdev =3D 0.056/0.058/0.185/0.011 ms ping with no-op nfqueue callback in C: root@lsrv:~# iptables -A OUTPUT -d 10.182.122.11/32 -j NFQUEUE = --queue-num 0 root@lsrv:~/nfqueue# ping -c 500 -i 0.01 -q 10.182.122.11 500 packets transmitted, 500 received, 0% packet loss, time 7981ms rtt min/avg/max/mdev =3D 0.117/0.123/0.384/0.020 ms ping with no-op nfqueue callback in Go: root@lsrv:~# iptables -A OUTPUT -d 10.182.122.11/32 -j NFQUEUE = --queue-num 0 root@lsrv:~# ping -c 500 -i 0.01 -q 10.182.122.11 500 packets transmitted, 500 received, 0% packet loss, time 7982ms rtt min/avg/max/mdev =3D 0.095/0.172/0.532/0.042 ms The mean induced latency of 65us for C or 114us for Go might be within = your parameters, except you mentioned 10us for WiFi bus arbitration, = which does indeed look impossible with this setup, even in C. Iperf3 results: iperf3 without nfqueue: root@lsrv:~# iptables -F OUTPUT root@lsrv:~# iperf3 -t 5 -c 10.182.122.11 Connecting to host 10.182.122.11, port 5201 [ 4] local 10.182.122.1 port 55810 connected to 10.182.122.11 port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 452 MBytes 3.79 Gbits/sec 0 178 KBytes = =20 [ 4] 1.00-2.00 sec 454 MBytes 3.82 Gbits/sec 0 320 KBytes = =20 [ 4] 2.00-3.00 sec 450 MBytes 3.77 Gbits/sec 0 320 KBytes = =20 [ 4] 3.00-4.00 sec 451 MBytes 3.79 Gbits/sec 0 352 KBytes = =20 [ 4] 4.00-5.00 sec 451 MBytes 3.79 Gbits/sec 0 352 KBytes = =20 - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-5.00 sec 2.21 GBytes 3.79 Gbits/sec 0 = sender [ 4] 0.00-5.00 sec 2.21 GBytes 3.79 Gbits/sec = receiver iperf Done. iperf3 with no-op nfqueue callback in C: root@lsrv:~# iptables -A OUTPUT -d 10.182.122.11/32 -j NFQUEUE = --queue-num 0 root@lsrv:~/nfqueue# iperf3 -t 5 -c 10.182.122.11 Connecting to host 10.182.122.11, port 5201 [ 4] local 10.182.122.1 port 55868 connected to 10.182.122.11 port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 17.4 MBytes 146 Mbits/sec 0 107 KBytes = =20 [ 4] 1.00-2.00 sec 16.9 MBytes 142 Mbits/sec 0 107 KBytes = =20 [ 4] 2.00-3.00 sec 17.0 MBytes 142 Mbits/sec 0 107 KBytes = =20 [ 4] 3.00-4.00 sec 17.0 MBytes 142 Mbits/sec 0 107 KBytes = =20 [ 4] 4.00-5.00 sec 17.0 MBytes 143 Mbits/sec 0 115 KBytes = =20 - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-5.00 sec 85.3 MBytes 143 Mbits/sec 0 = sender [ 4] 0.00-5.00 sec 84.7 MBytes 142 Mbits/sec = receiver iperf3 with no-op nfqueue callback in Go: root@lsrv:~# iptables -A OUTPUT -d 10.182.122.11/32 -j NFQUEUE = --queue-num 0 root@lsrv:~# iperf3 -t 5 -c 10.182.122.11 Connecting to host 10.182.122.11, port 5201 [ 4] local 10.182.122.1 port 55864 connected to 10.182.122.11 port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 14.6 MBytes 122 Mbits/sec 0 96.2 KBytes = =20 [ 4] 1.00-2.00 sec 14.1 MBytes 118 Mbits/sec 0 96.2 KBytes = =20 [ 4] 2.00-3.00 sec 14.0 MBytes 118 Mbits/sec 0 102 KBytes = =20 [ 4] 3.00-4.00 sec 14.0 MBytes 117 Mbits/sec 0 102 KBytes = =20 [ 4] 4.00-5.00 sec 13.7 MBytes 115 Mbits/sec 0 107 KBytes = =20 - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-5.00 sec 70.5 MBytes 118 Mbits/sec 0 = sender [ 4] 0.00-5.00 sec 69.9 MBytes 117 Mbits/sec = receiver iperf Done. So rats, throughput gets brutalized for both C and Go. For Go, a rate of = 117 Mbit with a 1500 byte MTU is 9750 packets/sec, which is 103us / = packet. Mean induced latency measured by ping is 114us, which is not far = off 103us, so the rate slowdown looks to be mostly caused by the = per-packet nfqueue calls. The core running test_nfqueue is pinned at = 100% during the test. "nice -n -20" does nothing. Presumably you=E2=80=99ll sometimes be releasing more than one packet at = a time(?) so I guess whether or not this is workable depends on how many = you release at once, what hardware you=E2=80=99re on and what rates you = need to test at. But when you=E2=80=99re trying to test a qdisc, I guess = you=E2=80=99d want to minimize the burden you add to the CPU, or else = move it to a core the qdisc isn=E2=80=99t running on, or something, so = the qdisc itself isn=E2=80=99t affected by the test rig. > There is of course a hideous amount of complexity moved to the daemon, I can only imagine. > as a pure fifo ap queue forms aggregregates much differently > than a fq_codeled one. But, yea! userspace.... This would be awesome if it works out! After that iperf3 test though, I = think I may have smashed my dreams of writing a libnetfilter_queue = userspace qdisc in Go, or C for that matter. If this does somehow turn out to be good enough performance-wise, I = think you=E2=80=99d have a lot more fun and spend a lot less time on it = in Go than C, but that=E2=80=99s just an opinion... :) --Apple-Mail=_D879296B-C415-4DB5-B5D9-A9D31D0F199B Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8
On Jun 16, 2018, at 12:30 AM, Dave Taht <dave.taht@gmail.com>= wrote:

Eric just suggested using the iptables NFQUEUE = ability to toss
packets to userspace.

https://home.regit.org/netfilter-en/using-nfqueue-and-libnetfil= ter_queue/

For wifi, at least, timings are not = hugely critical, a few hundred
usec is something userspace can handle = reasonably accurately. I like
very much being able to separate out = mcast and treat that correctly in
userspace, also. I did want to be below = 10usec (wifi "bus"
arbitration), which I am dubious = about....

Now as for an = implementation language? C++ C? Go? Python? The
condition of the = wrapper library for go leaves a bit to be desired
( https://github.com/chifflier/nfqueue-go ) and given a choice = I'd
MUCH rather use a go than a = C.

This sounds = cool... So for fun, I compared ping and iperf3 with no-op nfqueue = callbacks in both C and Go. As for the hardware setup, I used two lxc = containers (effectively just veth) on an APU2.

For the Go program, I used test_nfqueue from the = wrapper above (which yes, does need some work) and removed debugging / = logging.

For the C program I used = this:
I removed any per-packet printf = calls and compiled with "gcc sample-helloworld.c -o nfq -lnfnetlink = -lnetfilter_queue=E2=80=9D.

Ping = results:

ping without = nfqueue:
root@lsrv:~# iptables -F = OUTPUT
root@lsrv:~# ping -c 500 -i 0.01 -q 10.182.122.11
500 packets transmitted, 500 received, 0% packet loss, time = 7985ms
rtt min/avg/max/mdev =3D 0.056/0.058/0.185/0.011 = ms

ping with no-op nfqueue = callback in C:
root@lsrv:~# iptables -A OUTPUT -d = 10.182.122.11/32 -j NFQUEUE --queue-num 0
root@lsrv:~/nfqueue# = ping -c 500 -i 0.01 -q 10.182.122.11
500 packets = transmitted, 500 received, 0% packet loss, time 7981ms
rtt = min/avg/max/mdev =3D 0.117/0.123/0.384/0.020 ms

ping with no-op nfqueue callback in = Go:
root@lsrv:~# iptables -A OUTPUT -d = 10.182.122.11/32 -j NFQUEUE --queue-num 0
root@lsrv:~# = ping -c 500 -i 0.01 -q 10.182.122.11
500 packets = transmitted, 500 received, 0% packet loss, time 7982ms
rtt = min/avg/max/mdev =3D 0.095/0.172/0.532/0.042 ms

The mean induced latency of 65us for C = or 114us for Go might be within your parameters, except you mentioned = 10us for WiFi bus arbitration, which does indeed look impossible with = this setup, even in C.

Iperf3 results:

iperf3 without nfqueue:
root@lsrv:~# iptables -F = OUTPUT
root@lsrv:~# iperf3 -t 5 -c 10.182.122.11
Connecting to host 10.182.122.11, port 5201
[ =  4] local 10.182.122.1 port 55810 connected to 10.182.122.11 port = 5201
[ ID] Interval           = Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec   452 MBytes =  3.79 Gbits/sec    0    178 KBytes    =    
[  4]   1.00-2.00   sec =   454 MBytes  3.82 Gbits/sec    0    320 = KBytes       
[  4]   = 2.00-3.00   sec   450 MBytes  3.77 Gbits/sec   =  0    320 KBytes       
[=  4]   3.00-4.00   sec   451 MBytes  3.79 = Gbits/sec    0    352 KBytes      =  
[  4]   4.00-5.00   sec   451 = MBytes  3.79 Gbits/sec    0    352 KBytes  =      
- - - - - - - - - - - - - - - - - - - = - - - - - -
[ ID] Interval         =   Transfer     Bandwidth       Retr
[  4]   0.00-5.00   sec  2.21 GBytes =  3.79 Gbits/sec    0           =   sender
[  4]   0.00-5.00   sec =  2.21 GBytes  3.79 Gbits/sec         =          receiver
iperf = Done.

iperf3 with no-op nfqueue callback in C:
root@lsrv:~# iptables -A OUTPUT -d = 10.182.122.11/32 -j NFQUEUE --queue-num 0
root@lsrv:~/nfqueue# = iperf3 -t 5 -c 10.182.122.11
Connecting to host = 10.182.122.11, port 5201
[  4] local 10.182.122.1 = port 55868 connected to 10.182.122.11 port 5201
[ ID] = Interval           Transfer     = Bandwidth       Retr  Cwnd
[  4] =   0.00-1.00   sec  17.4 MBytes   146 Mbits/sec =    0    107 KBytes       
[  4]   1.00-2.00   sec  16.9 MBytes =   142 Mbits/sec    0    107 KBytes    =    
[  4]   2.00-3.00   sec =  17.0 MBytes   142 Mbits/sec    0    107 = KBytes       
[  4]   = 3.00-4.00   sec  17.0 MBytes   142 Mbits/sec   =  0    107 KBytes       
[=  4]   4.00-5.00   sec  17.0 MBytes   143 = Mbits/sec    0    115 KBytes      =  
- - - - - - - - - - - - - - - - - - - - - - - - = -
[ ID] Interval           = Transfer     Bandwidth       Retr
[=  4]   0.00-5.00   sec  85.3 MBytes   143 = Mbits/sec    0             = sender
[  4]   0.00-5.00   sec  84.7 = MBytes   142 Mbits/sec             =      receiver

iperf3 with no-op nfqueue callback in Go:
root@lsrv:~# iptables -A OUTPUT -d 10.182.122.11/32 -j = NFQUEUE --queue-num 0
root@lsrv:~# iperf3 -t 5 -c = 10.182.122.11
Connecting to host 10.182.122.11, port = 5201
[  4] local 10.182.122.1 port 55864 connected to = 10.182.122.11 port 5201
[ ID] Interval     =       Transfer     Bandwidth     =   Retr  Cwnd
[  4]   0.00-1.00   = sec  14.6 MBytes   122 Mbits/sec    0   96.2 = KBytes       
[  4]   = 1.00-2.00   sec  14.1 MBytes   118 Mbits/sec   =  0   96.2 KBytes       
[ =  4]   2.00-3.00   sec  14.0 MBytes   118 = Mbits/sec    0    102 KBytes      =  
[  4]   3.00-4.00   sec  14.0 = MBytes   117 Mbits/sec    0    102 KBytes  =      
[  4]   4.00-5.00   = sec  13.7 MBytes   115 Mbits/sec    0   =  107 KBytes       
- - - - - - - = - - - - - - - - - - - - - - - - - -
[ ID] Interval   =         Transfer     Bandwidth   =     Retr
[  4]   0.00-5.00   sec =  70.5 MBytes   118 Mbits/sec    0     =         sender
[  4]   = 0.00-5.00   sec  69.9 MBytes   117 Mbits/sec   =                receiver
iperf Done.

So rats, throughput gets brutalized for = both C and Go. For Go, a rate of 117 Mbit with a 1500 byte MTU is 9750 = packets/sec, which is 103us / packet. Mean induced latency measured by = ping is 114us, which is not far off 103us, so the rate slowdown looks to = be mostly caused by the per-packet nfqueue calls. The core running = test_nfqueue is pinned at 100% during the test. "nice -n -20" does = nothing.

Presumably you=E2=80=99ll sometimes be releasing more than = one packet at a time(?) so I guess whether or not this is workable = depends on how many you release at once, what hardware you=E2=80=99re on = and what rates you need to test at. But when you=E2=80=99re trying to = test a qdisc, I guess you=E2=80=99d want to minimize the burden you add = to the CPU, or else move it to a core the qdisc isn=E2=80=99t running = on, or something, so the qdisc itself isn=E2=80=99t affected by the test = rig.

There is of course a = hideous amount of complexity moved to the daemon,

I can only imagine.

as a pure fifo ap = queue forms aggregregates much differently
than a fq_codeled one. But, yea! = userspace....

This = would be awesome if it works out! After that iperf3 test though, I think = I may have smashed my dreams of writing a libnetfilter_queue userspace = qdisc in Go, or C for that matter.

If this does somehow turn out to be = good enough performance-wise, I think you=E2=80=99d have a lot more fun = and spend a lot less time on it in Go than C, but that=E2=80=99s just an = opinion... :)

= --Apple-Mail=_D879296B-C415-4DB5-B5D9-A9D31D0F199B--