From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-qt0-x242.google.com (mail-qt0-x242.google.com [IPv6:2607:f8b0:400d:c0d::242]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id DAF8F3BA8E for ; Sun, 17 Jun 2018 12:10:00 -0400 (EDT) Received: by mail-qt0-x242.google.com with SMTP id y31-v6so13240937qty.9 for ; Sun, 17 Jun 2018 09:10:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=ZZqK6kEpRA+8zz4LsTQN5seqXatqQ/C68enK+zL2SK4=; b=vU5eaXaTfi5dh6Fx8QK069xA1NgMvmDMIL7HHsYM8x+PeG6nNNBNwwcEb5Jo2lRkAe 2uFVadNNxvizT/ATJQIFtaYkFJNRk0e72SwpKxdiKxreOoeCv75oGB3Y7mOVTz9U1jWu BSTj1oCbXvZoztXewpoUHcdtkCgH0OphcwPIUNw6XSB9r4jF5imNN5jX7qSbXFT7MFHN b4dyne40vjlXS+46g3Z6byyHcYKua6hbir+SKUmSZH56jqHbQT1CDqYgOcHDHCik+9AA ttDAvZbCXS0PqgCmIZv9bvkN9ejYA63saw90s58h9stojNZiS5cZ5MST70XoLu2dToeo S40A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=ZZqK6kEpRA+8zz4LsTQN5seqXatqQ/C68enK+zL2SK4=; b=sE5e748s5dbth804jKD2TNjuHZtZaaw/a+p5raHNt0dk3UIyerBi0jf+NZXDr9QPkJ gw4URAe/j1XmdhroBsqJ4O1lNToznMIzEGk1TNqH6xqhNOsfuJ7YKDzMKEJbmPxfoSBe 1pM2//5wbPlQogfGH6F0uRZg3XKLLVeLgbDT5k2Jhh/9SGod6XWhJ3CahGjkmcHh0FMy QGxkF3Jt4ILVGASF9zJfjL4FwEue5pvScvKCmp20z2Cl0mCJ5xfBqhVSzK0IehaX5xSR Ke4HdAYHrFE0syTC73JRWg52kyl+Y1aolTvhmknXnEg5x6oP/r+LtEI5k8O3qt0dVZu9 QaDQ== X-Gm-Message-State: APt69E1jauxlmZZGXxE4LSnz29wdz/ZGjtDNxlTnlzGlArKH3NOkaNUV nZdtVWBKLeN9SyYY+ftv+DQW2KbrxiOLemWw8G0= X-Google-Smtp-Source: ADUXVKKzEZJT/KDiGOTenkCLYGOAWZ8pDRkY4V5Msc+aLAS7Bvb/vTpQJ5ua3R0wF3gkJIkLOm/FiBTiJAD2JPHIJXA= X-Received: by 2002:ac8:37c3:: with SMTP id e3-v6mr8274038qtc.199.1529251800325; Sun, 17 Jun 2018 09:10:00 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:aed:24f0:0:0:0:0:0 with HTTP; Sun, 17 Jun 2018 09:09:59 -0700 (PDT) In-Reply-To: References: <1527721073.171416827@apps.rackspace.com> <150ABF21-FAFC-48E2-9E55-CAA609EAE449@heistp.net> <20180617131921.09bf5353@redhat.com> From: Dave Taht Date: Sun, 17 Jun 2018 09:09:59 -0700 Message-ID: To: Pete Heist Cc: Jesper Dangaard Brouer , Florian Westphal , Marek Majkowski , Make-Wifi-fast Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Subject: Re: [Make-wifi-fast] emulating wifi better - coupling qdiscs in netem? X-BeenThere: make-wifi-fast@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 17 Jun 2018 16:10:01 -0000 Regrettably I consider most of the gains of gro/gso to be illusory for a router - in my ideal world packets are well distributed, and like jesper's 40+gigE work I care a lot (in terms of simulation baseline) about the overhead of a single ack packet in the wifi case. I'm pleased that these are apu2 results, I have some hope that a heftier box can do better. And these are not using the bulk verdict facility? Still it seems like keeping trying to couple qdiscs in netem in the kernel would be in the 6-10us range. If only... On Sun, Jun 17, 2018 at 8:16 AM, Pete Heist wrote: > Hi Jesper/Florian, thanks for noticing that, not surprisingly it doesn=E2= =80=99t > change the ping results much, but it improves throughput a lot (now only > ~20% less than without nfqueue): > > root@lsrv:~# iperf3 -t 5 -c 10.182.122.11 > Connecting to host 10.182.122.11, port 5201 > [ 4] local 10.182.122.1 port 55936 connected to 10.182.122.11 port 5201 > [ ID] Interval Transfer Bandwidth Retr Cwnd > [ 4] 0.00-1.00 sec 375 MBytes 3.14 Gbits/sec 173 372 KBytes > [ 4] 1.00-2.00 sec 365 MBytes 3.06 Gbits/sec 316 382 KBytes > [ 4] 2.00-3.00 sec 372 MBytes 3.13 Gbits/sec 368 427 KBytes > [ 4] 3.00-4.00 sec 364 MBytes 3.05 Gbits/sec 137 402 KBytes > [ 4] 4.00-5.00 sec 364 MBytes 3.05 Gbits/sec 342 382 KBytes > - - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval Transfer Bandwidth Retr > [ 4] 0.00-5.00 sec 1.80 GBytes 3.09 Gbits/sec 1336 > sender > [ 4] 0.00-5.00 sec 1.79 GBytes 3.08 Gbits/sec > receiver > iperf Done. > > I don=E2=80=99t know if/how the use of GSO affects Dave=E2=80=99s simulat= ion work, but I=E2=80=99ll > leave that to him. I only wanted to contribute a quick evaluation. :) > > Pete > > On Jun 17, 2018, at 1:19 PM, Jesper Dangaard Brouer > wrote: > > > Hi Pete, > > Happened to be at the Netfilter Workshop, and discussed nfqueue with > Florian and Marek, and I saw this attempt to use nfqueue, and Florian > points out that you are not using the GRO facility of nfqueue. > > I'll quote what Florian said below: > > On Sun, 17 Jun 2018 12:45:52 +0200 Florian Westphal wrote: > > The linked example code is old and does not set > mnl_attr_put_u32(nlh, NFQA_CFG_FLAGS, htonl(NFQA_CFG_F_GSO)); > > When requesting the queue. > > This means kernel has to do software segmentation of GSO skbs. > > Consider using > https://git.netfilter.org/libnetfilter_queue/tree/examples/nf-queue.c > > instead if you need a template, it does this correctly. > > > --Jesper > > > On Sun, 17 Jun 2018 00:53:03 +0200 Pete Heist wrote: > > On Jun 16, 2018, at 12:30 AM, Dave Taht wrote: > > Eric just suggested using the iptables NFQUEUE ability to toss > packets to userspace. > > https://home.regit.org/netfilter-en/using-nfqueue-and-libnetfilter_queue/ > > For wifi, at least, timings are not hugely critical, a few hundred > usec is something userspace can handle reasonably accurately. I like > very much being able to separate out mcast and treat that correctly in > userspace, also. I did want to be below 10usec (wifi "bus" > arbitration), which I am dubious about.... > > Now as for an implementation language? C++ C? Go? Python? The > condition of the wrapper library for go leaves a bit to be desired > ( https://github.com/chifflier/nfqueue-go > ) and given a choice I'd > MUCH rather use a go than a C. > > > This sounds cool... So for fun, I compared ping and iperf3 with no-op > nfqueue callbacks in both C and Go. As for the hardware setup, I used two > lxc containers (effectively just veth) on an APU2. > > For the Go program, I used test_nfqueue from the wrapper above (which yes= , > does need some work) and removed debugging / logging. > > For the C program I used this: > https://github.com/irontec/netfilter-nfqueue-samples/blob/master/sample-h= elloworld.c > I removed any per-packet printf calls and compiled with "gcc > sample-helloworld.c -o nfq -lnfnetlink -lnetfilter_queue=E2=80=9D. > > Ping results: > > ping without nfqueue: > root@lsrv:~# iptables -F OUTPUT > root@lsrv:~# ping -c 500 -i 0.01 -q 10.182.122.11 > 500 packets transmitted, 500 received, 0% packet loss, time 7985ms > rtt min/avg/max/mdev =3D 0.056/0.058/0.185/0.011 ms > > ping with no-op nfqueue callback in C: > root@lsrv:~# iptables -A OUTPUT -d 10.182.122.11/32 -j NFQUEUE --queue-nu= m 0 > root@lsrv:~/nfqueue# ping -c 500 -i 0.01 -q 10.182.122.11 > 500 packets transmitted, 500 received, 0% packet loss, time 7981ms > rtt min/avg/max/mdev =3D 0.117/0.123/0.384/0.020 ms > > ping with no-op nfqueue callback in Go: > root@lsrv:~# iptables -A OUTPUT -d 10.182.122.11/32 -j NFQUEUE --queue-nu= m 0 > root@lsrv:~# ping -c 500 -i 0.01 -q 10.182.122.11 > 500 packets transmitted, 500 received, 0% packet loss, time 7982ms > rtt min/avg/max/mdev =3D 0.095/0.172/0.532/0.042 ms > > The mean induced latency of 65us for C or 114us for Go might be within yo= ur > parameters, except you mentioned 10us for WiFi bus arbitration, which doe= s > indeed look impossible with this setup, even in C. > > Iperf3 results: > > iperf3 without nfqueue: > root@lsrv:~# iptables -F OUTPUT > root@lsrv:~# iperf3 -t 5 -c 10.182.122.11 > Connecting to host 10.182.122.11, port 5201 > [ 4] local 10.182.122.1 port 55810 connected to 10.182.122.11 port 5201 > [ ID] Interval Transfer Bandwidth Retr Cwnd > [ 4] 0.00-1.00 sec 452 MBytes 3.79 Gbits/sec 0 178 KBytes > [ 4] 1.00-2.00 sec 454 MBytes 3.82 Gbits/sec 0 320 KBytes > [ 4] 2.00-3.00 sec 450 MBytes 3.77 Gbits/sec 0 320 KBytes > [ 4] 3.00-4.00 sec 451 MBytes 3.79 Gbits/sec 0 352 KBytes > [ 4] 4.00-5.00 sec 451 MBytes 3.79 Gbits/sec 0 352 KBytes > - - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval Transfer Bandwidth Retr > [ 4] 0.00-5.00 sec 2.21 GBytes 3.79 Gbits/sec 0 sen= der > [ 4] 0.00-5.00 sec 2.21 GBytes 3.79 Gbits/sec > receiver > iperf Done. > > iperf3 with no-op nfqueue callback in C: > root@lsrv:~# iptables -A OUTPUT -d 10.182.122.11/32 -j NFQUEUE --queue-nu= m 0 > root@lsrv:~/nfqueue# iperf3 -t 5 -c 10.182.122.11 > Connecting to host 10.182.122.11, port 5201 > [ 4] local 10.182.122.1 port 55868 connected to 10.182.122.11 port 5201 > [ ID] Interval Transfer Bandwidth Retr Cwnd > [ 4] 0.00-1.00 sec 17.4 MBytes 146 Mbits/sec 0 107 KBytes > [ 4] 1.00-2.00 sec 16.9 MBytes 142 Mbits/sec 0 107 KBytes > [ 4] 2.00-3.00 sec 17.0 MBytes 142 Mbits/sec 0 107 KBytes > [ 4] 3.00-4.00 sec 17.0 MBytes 142 Mbits/sec 0 107 KBytes > [ 4] 4.00-5.00 sec 17.0 MBytes 143 Mbits/sec 0 115 KBytes > - - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval Transfer Bandwidth Retr > [ 4] 0.00-5.00 sec 85.3 MBytes 143 Mbits/sec 0 sen= der > [ 4] 0.00-5.00 sec 84.7 MBytes 142 Mbits/sec > receiver > > iperf3 with no-op nfqueue callback in Go: > root@lsrv:~# iptables -A OUTPUT -d 10.182.122.11/32 -j NFQUEUE --queue-nu= m 0 > root@lsrv:~# iperf3 -t 5 -c 10.182.122.11 > Connecting to host 10.182.122.11, port 5201 > [ 4] local 10.182.122.1 port 55864 connected to 10.182.122.11 port 5201 > [ ID] Interval Transfer Bandwidth Retr Cwnd > [ 4] 0.00-1.00 sec 14.6 MBytes 122 Mbits/sec 0 96.2 KBytes > [ 4] 1.00-2.00 sec 14.1 MBytes 118 Mbits/sec 0 96.2 KBytes > [ 4] 2.00-3.00 sec 14.0 MBytes 118 Mbits/sec 0 102 KBytes > [ 4] 3.00-4.00 sec 14.0 MBytes 117 Mbits/sec 0 102 KBytes > [ 4] 4.00-5.00 sec 13.7 MBytes 115 Mbits/sec 0 107 KBytes > - - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval Transfer Bandwidth Retr > [ 4] 0.00-5.00 sec 70.5 MBytes 118 Mbits/sec 0 sen= der > [ 4] 0.00-5.00 sec 69.9 MBytes 117 Mbits/sec > receiver > iperf Done. > > So rats, throughput gets brutalized for both C and Go. For Go, a rate of = 117 > Mbit with a 1500 byte MTU is 9750 packets/sec, which is 103us / packet. M= ean > induced latency measured by ping is 114us, which is not far off 103us, so > the rate slowdown looks to be mostly caused by the per-packet nfqueue cal= ls. > The core running test_nfqueue is pinned at 100% during the test. "nice -n > -20" does nothing. > > Presumably you=E2=80=99ll sometimes be releasing more than one packet at = a time(?) > so I guess whether or not this is workable depends on how many you releas= e > at once, what hardware you=E2=80=99re on and what rates you need to test = at. But > when you=E2=80=99re trying to test a qdisc, I guess you=E2=80=99d want to= minimize the > burden you add to the CPU, or else move it to a core the qdisc isn=E2=80= =99t running > on, or something, so the qdisc itself isn=E2=80=99t affected by the test = rig. > > There is of course a hideous amount of complexity moved to the daemon, > > > I can only imagine. > > as a pure fifo ap queue forms aggregregates much differently > than a fq_codeled one. But, yea! userspace.... > > > This would be awesome if it works out! After that iperf3 test though, I > think I may have smashed my dreams of writing a libnetfilter_queue usersp= ace > qdisc in Go, or C for that matter. > > If this does somehow turn out to be good enough performance-wise, I think > you=E2=80=99d have a lot more fun and spend a lot less time on it in Go t= han C, but > that=E2=80=99s just an opinion... :) > > > > > -- > Best regards, > Jesper Dangaard Brouer > MSc.CS, Principal Kernel Engineer at Red Hat > LinkedIn: http://www.linkedin.com/in/brouer > > --=20 Dave T=C3=A4ht CEO, TekLibre, LLC http://www.teklibre.com Tel: 1-669-226-2619