From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-lf1-x12d.google.com (mail-lf1-x12d.google.com [IPv6:2a00:1450:4864:20::12d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 64C803B2A4 for ; Fri, 30 Dec 2022 15:41:17 -0500 (EST) Received: by mail-lf1-x12d.google.com with SMTP id cf42so33023830lfb.1 for ; Fri, 30 Dec 2022 12:41:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=bj7NDEQATRYCSFAxJavBR0iJ6TE8htugoPaogoGTdTk=; b=R7SPjNiqZADBCWIeEpLff5Vc5euOfd2cY93fSw+85ViiflF/dh9+7s/jic36PU9MKy 2YjrRJaFELdjkP9fvdnYnJW1UbYocW1foVWKOj+Ew4/4Hwe3D2x8K9wnLaNJuGX2vVl5 7lgg/fizZlCTgWBDAsBCFBWdBfxXyzFTR9lDY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=bj7NDEQATRYCSFAxJavBR0iJ6TE8htugoPaogoGTdTk=; b=M8Cl0ZgU4bi1gIViUCdcxmh2IwW8Ng8Fh9MoEWGGU3XrxoUSlxKA4dtmGvLHD+eBex PGch/dBGXTScK6wmbGbdLoi3H8M5UxHZpphgQE1nfh88RcbO/fPSUBLZgwqk1bfJTQ62 opxTyz1JXOWJy3zq6E2gt/iMOuvAsMn9j6ouhQEpH/xVLLalwEfPk98U9Ulcxbutcw0H Y4nl6OGWsSRhLC8mkejh7bVOZ4X17IazbYekui38cy9Vtt3y7mIBAvTBk9vcNwWItDup /QeaQMlHtO2hm+u9120E70JUqyMkKTNvvtUQ6nOy4TIA3+F5y0JuHXRbWRlhrDes5U3M 0tRw== X-Gm-Message-State: AFqh2kqbEszEEckCejMBQMPEY/3pHgs9ZNlOTsCOGIg9S/lNDUSsJfPU vcBRZc+hVQ0FBI82QQOxH3LFb5aJMYgQ0UAU8n7rsxqBI6FW/cfw3CHEO90C5c7xPopqxZh4jay YzJihZ0LD5a+B3xQKTc8ZwJCm86cfnXqOmI1rCQWfEw== X-Google-Smtp-Source: AMrXdXs1W/LVU4dbh+nxrkJQhlxotMSc3BJ8HLyBHiblMlpLDEsbETS2pB0mzVy+lflG9I73hcHk/vyirQHsZTlsyJ4= X-Received: by 2002:a19:5045:0:b0:4b4:ee78:b8e5 with SMTP id z5-20020a195045000000b004b4ee78b8e5mr3289404lfj.293.1672432875223; Fri, 30 Dec 2022 12:41:15 -0800 (PST) MIME-Version: 1.0 References: <20221220075215.1.Ic12e347e0d61a618124b742614e82bbd5d770173@changeid> In-Reply-To: From: Bob McMahon Date: Fri, 30 Dec 2022 12:41:03 -0800 Message-ID: To: Dave Taht Cc: Abhishek Kumar , netdev@vger.kernel.org, kvalo@kernel.org, linux-wireless@vger.kernel.org, linux-kernel@vger.kernel.org, Eric Dumazet , Make-Wifi-fast , Jakub Kicinski , Paolo Abeni , "David S. Miller" Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha-256; boundary="00000000000078979705f1119f1c" Subject: Re: [Make-wifi-fast] [PATCH] ath10k: snoc: enable threaded napi on WCN3990 X-BeenThere: make-wifi-fast@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Dec 2022 20:41:17 -0000 --00000000000078979705f1119f1c Content-Type: multipart/alternative; boundary="00000000000070b9d205f1119f52" --00000000000070b9d205f1119f52 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable FYI, we're trying to add "simple" responsiveness features to the 2.1.9 release. Industry input is appreciated. We've done a lot of one way delay work as well as different traffic profiles. I'm at a stage of trying to show Broadcom mgmt that releasing iperf 2 as open source to skilled network engineers will enable the networking industry (& the best network silicon providers) in the entire e2e path. Some questions I get are: - Who really uses it? - Why can't other open source or commercially provided tools be used? - Why don't others help out much with the open source effort? - What are the key performance indicators that will influence engineers in the industry in an actionable way? - What are the futures per the networking community, e.g. test support for UDP QUIC? Thanks, Bob Bob On Thu, Dec 29, 2022 at 7:44 PM Dave Taht wrote: > thx for leaning in bob! (for those of you that don't know, he's the > iperf2 maintainer, and if you haven't updated your distro/os/test > environment to 2.1.8 that came out in august, yet there's all kinds of > nifty new stuff in there: https://sourceforge.net/projects/iperf2 - > too much to list, and the -i 1 -e option is just the start of it.... > > Iperf2's principal flaw is that people think iperf3 is the successor > to it, and no, they are very different codebases, with very different > features. iperf =3D iperf2, for most, but not all. > > On Wed, Dec 28, 2022 at 9:54 PM Bob McMahon > wrote: > > > > A bit of a tangent but iperf 2's enhanced option provides sampled RTT > and CWND in the interval output. I requires -e. It also provide the conne= ct > time, the icwnd, irtt and the mss per the connect. > > > > [rjmcmahon@rjm-nas ~]$ iperf -c 192.168.1.85 -i 1 -e > > ------------------------------------------------------------ > > Client connecting to 192.168.1.85, TCP port 5001 with pid 20041 (1 flow= s) > > Write buffer size: 131072 Byte > > TOS set to 0x0 (Nagle on) > > TCP window size: 16.0 KByte (default) > > ------------------------------------------------------------ > > [ 1] local 192.168.1.73%eno1 port 38822 connected with 192.168.1.85 > port 5001 (sock=3D3) (icwnd/mss/irtt=3D14/1448/246) (ct=3D0.32 ms) on 202= 2-12-28 > 21:51:13 (PST) > > [ ID] Interval Transfer Bandwidth Write/Err Rtry > Cwnd/RTT(var) NetPwr > > [ 1] 0.00-1.00 sec 114 MBytes 952 Mbits/sec 908/0 3 > 353K/1901(59) us 62606 > > [ 1] 1.00-2.00 sec 111 MBytes 932 Mbits/sec 889/0 0 > 354K/2029(115) us 57429 > > [ 1] 2.00-3.00 sec 112 MBytes 937 Mbits/sec 894/0 3 > 354K/2061(84) us 56855 > > [ 1] 3.00-4.00 sec 112 MBytes 938 Mbits/sec 895/0 3 > 354K/2027(115) us 57873 > > [ 1] 4.00-5.00 sec 112 MBytes 938 Mbits/sec 895/0 6 > 354K/2046(68) us 57336 > > [ 1] 5.00-6.00 sec 112 MBytes 937 Mbits/sec 894/0 0 > 374K/2160(109) us 54249 > > [ 1] 6.00-7.00 sec 111 MBytes 933 Mbits/sec 890/0 2 > 374K/2155(115) us 54132 > > [ 1] 7.00-8.00 sec 111 MBytes 934 Mbits/sec 891/0 2 > 374K/2139(120) us 54598 > > [ 1] 8.00-9.00 sec 112 MBytes 938 Mbits/sec 895/0 0 > 374K/2129(126) us 55101 > > [ 1] 9.00-10.00 sec 112 MBytes 935 Mbits/sec 892/0 1 > 374K/2200(102) us 53144 > > [ 1] 0.00-10.03 sec 1.09 GBytes 935 Mbits/sec 8944/0 20 > 374K/4082(4010) us 28632 > > This is a wonderful new feature, much simpler than relying on packet > captures. However! this test result above is on a decent, ethernet, > network, where on wifi, or anything bufferbloated, you would see the > cwnd and rtt inflate and/or bounce around horribly. > > > > > > > > Bob > > > > On Wed, Dec 28, 2022 at 4:49 PM Dave Taht via Make-wifi-fast < > make-wifi-fast@lists.bufferbloat.net> wrote: > >> > >> On Wed, Dec 28, 2022 at 3:53 PM Abhishek Kumar > wrote: > >> > > >> > Apologies for the late reply, Thanks Dave for your comment. My answe= r > is inline. > >> > > >> > On Tue, Dec 20, 2022 at 7:10 AM Dave Taht > wrote: > >> > > > >> > > I am always interested in flent.org tcp_nup, tcp_ndown, and rrul_b= e > >> > > tests on wifi hardware. In AP mode, especially, against a few > clients > >> > > in rtt_fair on the "ending the anomaly" test suite at the bottom o= f > >> > > this link: https://www.cs.kau.se/tohojo/airtime-fairness/ . Of > these, > >> > > it's trying to optimize bandwidth more fairly and keep latencies l= ow > >> > > when 4 or more stations are trying to transmit (in a world with 16 > or > >> > > more stations online), that increasingly bothers me the most. I'm > >> > > seeing 5+ seconds on some rtt_fair-like tests nowadays. > >> > I used testing using iperf and conductive setup and fetched the > >> > throughput data(mentioned below). > >> > > > >> > > I was also seeing huge simultaneous upload vs download disparities > on > >> > > the latest kernels, on various threads over here: > >> > > https://forum.openwrt.org/t/aql-and-the-ath10k-is-lovely/59002 and > >> > > more recently here: > >> > > > https://forum.openwrt.org/t/reducing-multiplexing-latencies-still-further= -in-wifi/133605 > >> > Interesting, thanks for the pointer and probably the Qualcomm team i= s > >> > aware of it. > >> > > > >> > > I don't understand why napi with the default budget (64) is even > >> > > needed on the ath10k, as a single txop takes a minimum of ~200us, > but > >> > > perhaps your patch will help. Still, measuring the TCP statistics > >> > > in-band would be nice to see. Some new tools are appearing that ca= n > do > >> > > this, Apple's goresponsiveness, crusader... that are simpler to us= e > >> > > than flent. > >> > Here are some of the additional raw data with and without threaded > napi: > >> > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D > >> > udp_rx(Without threaded NAPI) > >> > 435.98+-5.16 : Channel 44 > >> > 439.06+-0.66 : Channel 157 > >> > > >> > udp_rx(With threaded NAPI) > >> > 509.73+-41.03 : Channel 44 > >> > 549.97+-7.62 : Channel 157 > >> > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D > >> > udp_tx(Without threaded NAPI) > >> > 461.31+-0.69 : Channel 44 > >> > 461.46+-0.78 : Channel 157 > >> > > >> > udp_tx(With threaded NAPI) > >> > 459.20+-0.77 : Channel 44 > >> > 459.78+-1.08 : Channel 157 > >> > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D > >> > tcp_rx(Without threaded NAPI) > >> > 472.63+-2.35 : Channel 44 > >> > 469.29+-6.31 : Channel 157 > >> > > >> > tcp_rx(With threaded NAPI) > >> > 498.49+-2.44 : Channel 44 > >> > 541.14+-40.65 : Channel 157 > >> > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D > >> > tcp_tx(Without threaded NAPI) > >> > 317.34+-2.37 : Channel 44 > >> > 317.01+-2.56 : Channel 157 > >> > > >> > tcp_tx(With threaded NAPI) > >> > 371.34+-2.36 : Channel 44 > >> > 376.95+-9.40 : Channel 157 > >> > >> My concern is primarily with the induced tcp latency on this test. A > >> way to check that is to run wireshark on your test client driving the > >> test, capture the iperf traffic, and then plot the "Statistics->TCP > >> stream statistics for both throughput and rtt. Would it be possible > >> for you to do that and put up those plots somewhere? > >> > >> The worst case test is a tcp bidirectional test which I don't know if > >> older iperfs can do. (iperf2 has new bounceback and bidir tests) > >> > >> Ideally stuff going in either direction, would not look as horrible, > >> as it did, back in 2016, documented in this linuxplumbers presentation > >> here: > https://blog.linuxplumbersconf.org/2016/ocw/system/presentations/3963/ori= ginal/linuxplumbers_wifi_latency-3Nov.pdf > >> and discussed on lwn, here: https://lwn.net/Articles/705884/ > >> > >> I worry about folk achieving slightly better tcp throughput at the > >> expense of clobbering in-tcp-stream latency. Back then we were > >> shooting for no more than 40ms extra latency under load on this chip, > >> down from (unusable) seconds. Presently elsewhere, on other chips, > >> we're getting 8ms with stuff that's not in tree for the ath10k, there > >> is a slight cost in single stream throughput but when multiple streams > >> are in use, on multiple stations, things like web pages fly, > >> irrespective of load. > >> > >> > >> > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D > >> > > >> > > > >> > > On Tue, Dec 20, 2022 at 12:17 AM Abhishek Kumar < > kuabhs@chromium.org> wrote: > >> > > > > >> > > > NAPI poll can be done in threaded context along with soft irq > >> > > > context. Threaded context can be scheduled efficiently, thus > >> > > > creating less of bottleneck during Rx processing. This patch is > >> > > > to enable threaded NAPI on ath10k driver. > >> > > > > >> > > > Based on testing, it was observed that on WCN3990, the CPU0 > reaches > >> > > > 100% utilization when napi runs in softirq context. At the same > >> > > > time the other CPUs are at low consumption percentage. This > >> > > > does not allow device to reach its maximum throughput potential. > >> > > > After enabling threaded napi, CPU load is balanced across all CP= Us > >> > > > and following improvments were observed: > >> > > > - UDP_RX increase by ~22-25% > >> > > > - TCP_RX increase by ~15% > >> > > > > >> > > > Tested-on: WCN3990 hw1.0 SNOC WLAN.HL.3.2.2-00696-QCAHLSWMTPL-1 > >> > > > Signed-off-by: Abhishek Kumar > >> > > > --- > >> > > > > >> > > > drivers/net/wireless/ath/ath10k/core.c | 16 ++++++++++++++++ > >> > > > drivers/net/wireless/ath/ath10k/hw.h | 2 ++ > >> > > > drivers/net/wireless/ath/ath10k/snoc.c | 3 +++ > >> > > > 3 files changed, 21 insertions(+) > >> > > > > >> > > > diff --git a/drivers/net/wireless/ath/ath10k/core.c > b/drivers/net/wireless/ath/ath10k/core.c > >> > > > index 5eb131ab916fd..ee4b6ba508c81 100644 > >> > > > --- a/drivers/net/wireless/ath/ath10k/core.c > >> > > > +++ b/drivers/net/wireless/ath/ath10k/core.c > >> > > > @@ -100,6 +100,7 @@ static const struct ath10k_hw_params > ath10k_hw_params_list[] =3D { > >> > > > .hw_restart_disconnect =3D false, > >> > > > .use_fw_tx_credits =3D true, > >> > > > .delay_unmap_buffer =3D false, > >> > > > + .enable_threaded_napi =3D false, > >> > > > }, > >> > > > { > >> > > > .id =3D QCA988X_HW_2_0_VERSION, > >> > > > @@ -140,6 +141,7 @@ static const struct ath10k_hw_params > ath10k_hw_params_list[] =3D { > >> > > > .hw_restart_disconnect =3D false, > >> > > > .use_fw_tx_credits =3D true, > >> > > > .delay_unmap_buffer =3D false, > >> > > > + .enable_threaded_napi =3D false, > >> > > > }, > >> > > > { > >> > > > .id =3D QCA9887_HW_1_0_VERSION, > >> > > > @@ -181,6 +183,7 @@ static const struct ath10k_hw_params > ath10k_hw_params_list[] =3D { > >> > > > .hw_restart_disconnect =3D false, > >> > > > .use_fw_tx_credits =3D true, > >> > > > .delay_unmap_buffer =3D false, > >> > > > + .enable_threaded_napi =3D false, > >> > > > }, > >> > > > { > >> > > > .id =3D QCA6174_HW_3_2_VERSION, > >> > > > @@ -217,6 +220,7 @@ static const struct ath10k_hw_params > ath10k_hw_params_list[] =3D { > >> > > > .hw_restart_disconnect =3D false, > >> > > > .use_fw_tx_credits =3D true, > >> > > > .delay_unmap_buffer =3D false, > >> > > > + .enable_threaded_napi =3D false, > >> > > > }, > >> > > > { > >> > > > .id =3D QCA6174_HW_2_1_VERSION, > >> > > > @@ -257,6 +261,7 @@ static const struct ath10k_hw_params > ath10k_hw_params_list[] =3D { > >> > > > .hw_restart_disconnect =3D false, > >> > > > .use_fw_tx_credits =3D true, > >> > > > .delay_unmap_buffer =3D false, > >> > > > + .enable_threaded_napi =3D false, > >> > > > }, > >> > > > { > >> > > > .id =3D QCA6174_HW_2_1_VERSION, > >> > > > @@ -297,6 +302,7 @@ static const struct ath10k_hw_params > ath10k_hw_params_list[] =3D { > >> > > > .hw_restart_disconnect =3D false, > >> > > > .use_fw_tx_credits =3D true, > >> > > > .delay_unmap_buffer =3D false, > >> > > > + .enable_threaded_napi =3D false, > >> > > > }, > >> > > > { > >> > > > .id =3D QCA6174_HW_3_0_VERSION, > >> > > > @@ -337,6 +343,7 @@ static const struct ath10k_hw_params > ath10k_hw_params_list[] =3D { > >> > > > .hw_restart_disconnect =3D false, > >> > > > .use_fw_tx_credits =3D true, > >> > > > .delay_unmap_buffer =3D false, > >> > > > + .enable_threaded_napi =3D false, > >> > > > }, > >> > > > { > >> > > > .id =3D QCA6174_HW_3_2_VERSION, > >> > > > @@ -381,6 +388,7 @@ static const struct ath10k_hw_params > ath10k_hw_params_list[] =3D { > >> > > > .hw_restart_disconnect =3D false, > >> > > > .use_fw_tx_credits =3D true, > >> > > > .delay_unmap_buffer =3D false, > >> > > > + .enable_threaded_napi =3D false, > >> > > > }, > >> > > > { > >> > > > .id =3D QCA99X0_HW_2_0_DEV_VERSION, > >> > > > @@ -427,6 +435,7 @@ static const struct ath10k_hw_params > ath10k_hw_params_list[] =3D { > >> > > > .hw_restart_disconnect =3D false, > >> > > > .use_fw_tx_credits =3D true, > >> > > > .delay_unmap_buffer =3D false, > >> > > > + .enable_threaded_napi =3D false, > >> > > > }, > >> > > > { > >> > > > .id =3D QCA9984_HW_1_0_DEV_VERSION, > >> > > > @@ -480,6 +489,7 @@ static const struct ath10k_hw_params > ath10k_hw_params_list[] =3D { > >> > > > .hw_restart_disconnect =3D false, > >> > > > .use_fw_tx_credits =3D true, > >> > > > .delay_unmap_buffer =3D false, > >> > > > + .enable_threaded_napi =3D false, > >> > > > }, > >> > > > { > >> > > > .id =3D QCA9888_HW_2_0_DEV_VERSION, > >> > > > @@ -530,6 +540,7 @@ static const struct ath10k_hw_params > ath10k_hw_params_list[] =3D { > >> > > > .hw_restart_disconnect =3D false, > >> > > > .use_fw_tx_credits =3D true, > >> > > > .delay_unmap_buffer =3D false, > >> > > > + .enable_threaded_napi =3D false, > >> > > > }, > >> > > > { > >> > > > .id =3D QCA9377_HW_1_0_DEV_VERSION, > >> > > > @@ -570,6 +581,7 @@ static const struct ath10k_hw_params > ath10k_hw_params_list[] =3D { > >> > > > .hw_restart_disconnect =3D false, > >> > > > .use_fw_tx_credits =3D true, > >> > > > .delay_unmap_buffer =3D false, > >> > > > + .enable_threaded_napi =3D false, > >> > > > }, > >> > > > { > >> > > > .id =3D QCA9377_HW_1_1_DEV_VERSION, > >> > > > @@ -612,6 +624,7 @@ static const struct ath10k_hw_params > ath10k_hw_params_list[] =3D { > >> > > > .hw_restart_disconnect =3D false, > >> > > > .use_fw_tx_credits =3D true, > >> > > > .delay_unmap_buffer =3D false, > >> > > > + .enable_threaded_napi =3D false, > >> > > > }, > >> > > > { > >> > > > .id =3D QCA9377_HW_1_1_DEV_VERSION, > >> > > > @@ -645,6 +658,7 @@ static const struct ath10k_hw_params > ath10k_hw_params_list[] =3D { > >> > > > .hw_restart_disconnect =3D false, > >> > > > .use_fw_tx_credits =3D true, > >> > > > .delay_unmap_buffer =3D false, > >> > > > + .enable_threaded_napi =3D false, > >> > > > }, > >> > > > { > >> > > > .id =3D QCA4019_HW_1_0_DEV_VERSION, > >> > > > @@ -692,6 +706,7 @@ static const struct ath10k_hw_params > ath10k_hw_params_list[] =3D { > >> > > > .hw_restart_disconnect =3D false, > >> > > > .use_fw_tx_credits =3D true, > >> > > > .delay_unmap_buffer =3D false, > >> > > > + .enable_threaded_napi =3D false, > >> > > > }, > >> > > > { > >> > > > .id =3D WCN3990_HW_1_0_DEV_VERSION, > >> > > > @@ -725,6 +740,7 @@ static const struct ath10k_hw_params > ath10k_hw_params_list[] =3D { > >> > > > .hw_restart_disconnect =3D true, > >> > > > .use_fw_tx_credits =3D false, > >> > > > .delay_unmap_buffer =3D true, > >> > > > + .enable_threaded_napi =3D true, > >> > > > }, > >> > > > }; > >> > > > > >> > > > diff --git a/drivers/net/wireless/ath/ath10k/hw.h > b/drivers/net/wireless/ath/ath10k/hw.h > >> > > > index 9643031a4427a..adf3076b96503 100644 > >> > > > --- a/drivers/net/wireless/ath/ath10k/hw.h > >> > > > +++ b/drivers/net/wireless/ath/ath10k/hw.h > >> > > > @@ -639,6 +639,8 @@ struct ath10k_hw_params { > >> > > > bool use_fw_tx_credits; > >> > > > > >> > > > bool delay_unmap_buffer; > >> > > > + > >> > > > + bool enable_threaded_napi; > >> > > > }; > >> > > > > >> > > > struct htt_resp; > >> > > > diff --git a/drivers/net/wireless/ath/ath10k/snoc.c > b/drivers/net/wireless/ath/ath10k/snoc.c > >> > > > index cfcb759a87dea..b94150fb6ef06 100644 > >> > > > --- a/drivers/net/wireless/ath/ath10k/snoc.c > >> > > > +++ b/drivers/net/wireless/ath/ath10k/snoc.c > >> > > > @@ -927,6 +927,9 @@ static int ath10k_snoc_hif_start(struct > ath10k *ar) > >> > > > > >> > > > bitmap_clear(ar_snoc->pending_ce_irqs, 0, CE_COUNT_MAX); > >> > > > > >> > > > + if (ar->hw_params.enable_threaded_napi) > >> > > > + dev_set_threaded(&ar->napi_dev, true); > >> > > > + > >> > > > ath10k_core_napi_enable(ar); > >> > > > ath10k_snoc_irq_enable(ar); > >> > > > ath10k_snoc_rx_post(ar); > >> > > > -- > >> > > > 2.39.0.314.g84b9a713c41-goog > >> > > > > >> > > > >> > > > >> > > -- > >> > > This song goes out to all the folk that thought Stadia would work: > >> > > > https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-698136666= 5607352320-FXtz > >> > > Dave T=C3=A4ht CEO, TekLibre, LLC > >> > >> > >> > >> -- > >> This song goes out to all the folk that thought Stadia would work: > >> > https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-698136666= 5607352320-FXtz > >> Dave T=C3=A4ht CEO, TekLibre, LLC > >> _______________________________________________ > >> Make-wifi-fast mailing list > >> Make-wifi-fast@lists.bufferbloat.net > >> https://lists.bufferbloat.net/listinfo/make-wifi-fast > > > > > > This electronic communication and the information and any files > transmitted with it, or attached to it, are confidential and are intended > solely for the use of the individual or entity to whom it is addressed an= d > may contain information that is confidential, legally privileged, protect= ed > by privacy laws, or otherwise restricted from disclosure to anyone else. = If > you are not the intended recipient or the person responsible for deliveri= ng > the e-mail to the intended recipient, you are hereby notified that any us= e, > copying, distributing, dissemination, forwarding, printing, or copying of > this e-mail is strictly prohibited. If you received this e-mail in error, > please return the e-mail to the sender, delete it from your computer, and > destroy any printed copy of it. > > > > -- > This song goes out to all the folk that thought Stadia would work: > > https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-698136666= 5607352320-FXtz > Dave T=C3=A4ht CEO, TekLibre, LLC > --=20 This electronic communication and the information and any files transmitted= =20 with it, or attached to it, are confidential and are intended solely for=20 the use of the individual or entity to whom it is addressed and may contain= =20 information that is confidential, legally privileged, protected by privacy= =20 laws, or otherwise restricted from disclosure to anyone else. If you are=20 not the intended recipient or the person responsible for delivering the=20 e-mail to the intended recipient, you are hereby notified that any use,=20 copying, distributing, dissemination, forwarding, printing, or copying of= =20 this e-mail is strictly prohibited. If you received this e-mail in error,= =20 please return the e-mail to the sender, delete it from your computer, and= =20 destroy any printed copy of it. --00000000000070b9d205f1119f52 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
FYI, we're trying to add "simple" responsive= ness features to the 2.1.9 release. Industry input is appreciated. We'v= e done a lot of one way delay work as well as different traffic profiles.
I'm at a stage of trying to show Broadcom mgmt that releasing ipe= rf 2 as open source to skilled network engineers will enable the networking= industry (& the best network silicon providers) in the entire e2e path= .

Some questions=C2=A0I get are:
  • Who really uses it?
  • Why can't other open source or commercially provided tools be use= d?
  • Why don't others help out much with the open source effort?= =C2=A0
  • What are the key performance indicators that will influence = engineers in the=C2=A0industry=C2=A0in an actionable way?
  • What are = the futures per the networking community, e.g. test support for UDP QUIC?
Thanks,
Bob

Bob

On Thu, Dec 29, 2022= at 7:44 PM Dave Taht <dave.taht@= gmail.com> wrote:
thx for leaning in bob! (for those of you that don't know, h= e's the
iperf2 maintainer, and if you haven't updated your distro/os/test
environment to 2.1.8 that came out in august, yet there's all kinds of<= br> nifty new stuff in there: https://sourceforge.net/projects/ip= erf2 -
too much to list, and the -i 1 -e option is just the start of it....

Iperf2's principal flaw is that people think iperf3 is the successor to it, and no, they are very different codebases, with very different
features. iperf =3D iperf2, for most, but not all.

On Wed, Dec 28, 2022 at 9:54 PM Bob McMahon <bob.mcmahon@broadcom.com> wrote:<= br> >
> A bit of a tangent but iperf 2's enhanced option provides sampled = RTT and CWND in the interval output. I requires -e. It also provide the con= nect time, the icwnd, irtt and the mss per the connect.
>
> [rjmcmahon@rjm-nas ~]$ iperf -c 192.168.1.85 -i 1 -e
> ------------------------------------------------------------
> Client connecting to 192.168.1.85, TCP port 5001 with pid 20041 (1 flo= ws)
> Write buffer size: 131072 Byte
> TOS set to 0x0 (Nagle on)
> TCP window size: 16.0 KByte (default)
> ------------------------------------------------------------
> [=C2=A0 1] local 192.168.1.73%eno1 port 38822 connected with 192.168.1= .85 port 5001 (sock=3D3) (icwnd/mss/irtt=3D14/1448/246) (ct=3D0.32 ms) on 2= 022-12-28 21:51:13 (PST)
> [ ID] Interval=C2=A0 =C2=A0 =C2=A0 =C2=A0 Transfer=C2=A0 =C2=A0 Bandwi= dth=C2=A0 =C2=A0 =C2=A0 =C2=A0Write/Err=C2=A0 Rtry=C2=A0 =C2=A0 =C2=A0Cwnd/= RTT(var)=C2=A0 =C2=A0 =C2=A0 =C2=A0 NetPwr
> [=C2=A0 1] 0.00-1.00 sec=C2=A0 =C2=A0114 MBytes=C2=A0 =C2=A0952 Mbits/= sec=C2=A0 908/0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A03=C2=A0 =C2=A0 =C2=A0 353K= /1901(59) us=C2=A0 62606
> [=C2=A0 1] 1.00-2.00 sec=C2=A0 =C2=A0111 MBytes=C2=A0 =C2=A0932 Mbits/= sec=C2=A0 889/0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A0 354K= /2029(115) us=C2=A0 57429
> [=C2=A0 1] 2.00-3.00 sec=C2=A0 =C2=A0112 MBytes=C2=A0 =C2=A0937 Mbits/= sec=C2=A0 894/0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A03=C2=A0 =C2=A0 =C2=A0 354K= /2061(84) us=C2=A0 56855
> [=C2=A0 1] 3.00-4.00 sec=C2=A0 =C2=A0112 MBytes=C2=A0 =C2=A0938 Mbits/= sec=C2=A0 895/0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A03=C2=A0 =C2=A0 =C2=A0 354K= /2027(115) us=C2=A0 57873
> [=C2=A0 1] 4.00-5.00 sec=C2=A0 =C2=A0112 MBytes=C2=A0 =C2=A0938 Mbits/= sec=C2=A0 895/0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A06=C2=A0 =C2=A0 =C2=A0 354K= /2046(68) us=C2=A0 57336
> [=C2=A0 1] 5.00-6.00 sec=C2=A0 =C2=A0112 MBytes=C2=A0 =C2=A0937 Mbits/= sec=C2=A0 894/0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A0 374K= /2160(109) us=C2=A0 54249
> [=C2=A0 1] 6.00-7.00 sec=C2=A0 =C2=A0111 MBytes=C2=A0 =C2=A0933 Mbits/= sec=C2=A0 890/0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A02=C2=A0 =C2=A0 =C2=A0 374K= /2155(115) us=C2=A0 54132
> [=C2=A0 1] 7.00-8.00 sec=C2=A0 =C2=A0111 MBytes=C2=A0 =C2=A0934 Mbits/= sec=C2=A0 891/0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A02=C2=A0 =C2=A0 =C2=A0 374K= /2139(120) us=C2=A0 54598
> [=C2=A0 1] 8.00-9.00 sec=C2=A0 =C2=A0112 MBytes=C2=A0 =C2=A0938 Mbits/= sec=C2=A0 895/0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A0 374K= /2129(126) us=C2=A0 55101
> [=C2=A0 1] 9.00-10.00 sec=C2=A0 =C2=A0112 MBytes=C2=A0 =C2=A0935 Mbits= /sec=C2=A0 892/0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A01=C2=A0 =C2=A0 =C2=A0 374= K/2200(102) us=C2=A0 53144
> [=C2=A0 1] 0.00-10.03 sec=C2=A0 1.09 GBytes=C2=A0 =C2=A0935 Mbits/sec= =C2=A0 8944/0=C2=A0 =C2=A0 =C2=A0 =C2=A0 20=C2=A0 =C2=A0 =C2=A0 374K/4082(4= 010) us=C2=A0 28632

This is a wonderful new feature, much simpler than relying on packet
captures. However! this test result above is on a decent, ethernet,
network, where on wifi, or anything bufferbloated, you would see the
cwnd and rtt inflate and/or bounce around horribly.


>
>
> Bob
>
> On Wed, Dec 28, 2022 at 4:49 PM Dave Taht via Make-wifi-fast <make-w= ifi-fast@lists.bufferbloat.net> wrote:
>>
>> On Wed, Dec 28, 2022 at 3:53 PM Abhishek Kumar <kuabhs@chromium.org> wrote= :
>> >
>> > Apologies for the late reply, Thanks Dave for your comment. M= y answer is inline.
>> >
>> > On Tue, Dec 20, 2022 at 7:10 AM Dave Taht <dave.taht@gmail.com> wrote= :
>> > >
>> > > I am always interested in flent.org tcp_nup, tcp_ndown, and r= rul_be
>> > > tests on wifi hardware. In AP mode, especially, against = a few clients
>> > > in rtt_fair on the "ending the anomaly" test s= uite at the bottom of
>> > > this link: https://www.cs.kau.se/to= hojo/airtime-fairness/ . Of these,
>> > > it's trying to optimize bandwidth more fairly and ke= ep latencies low
>> > > when 4 or more stations are trying to transmit (in a wor= ld with 16 or
>> > > more stations online), that increasingly bothers me the = most. I'm
>> > > seeing 5+ seconds on some rtt_fair-like tests nowadays.<= br> >> > I used testing using iperf and conductive setup and fetched t= he
>> > throughput data(mentioned below).
>> > >
>> > > I was also seeing huge simultaneous upload vs download d= isparities on
>> > > the latest kernels, on various threads over here:
>> > > https://forum.openw= rt.org/t/aql-and-the-ath10k-is-lovely/59002 and
>> > > more recently here:
>> > > https://forum.openwrt.org/t/reducing-multiplexing-latencies-still-fu= rther-in-wifi/133605
>> > Interesting, thanks for the pointer and probably the Qualcomm= team is
>> > aware of it.
>> > >
>> > > I don't understand why napi with the default budget = (64) is even
>> > > needed on the ath10k, as a single txop takes a minimum o= f ~200us, but
>> > > perhaps your patch will help. Still, measuring the TCP s= tatistics
>> > > in-band would be nice to see. Some new tools are appeari= ng that can do
>> > > this, Apple's goresponsiveness, crusader... that are= simpler to use
>> > > than flent.
>> > Here are some of the additional raw data with and without thr= eaded napi:
>> > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D
>> > udp_rx(Without threaded NAPI)
>> > 435.98+-5.16 : Channel 44
>> > 439.06+-0.66 : Channel 157
>> >
>> > udp_rx(With threaded NAPI)
>> > 509.73+-41.03 : Channel 44
>> > 549.97+-7.62 : Channel 157
>> > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D
>> > udp_tx(Without threaded NAPI)
>> > 461.31+-0.69=C2=A0 : Channel 44
>> > 461.46+-0.78 : Channel 157
>> >
>> > udp_tx(With threaded NAPI)
>> > 459.20+-0.77 : Channel 44
>> > 459.78+-1.08 : Channel 157
>> > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D
>> > tcp_rx(Without threaded NAPI)
>> > 472.63+-2.35 : Channel 44
>> > 469.29+-6.31 : Channel 157
>> >
>> > tcp_rx(With threaded NAPI)
>> > 498.49+-2.44 : Channel 44
>> > 541.14+-40.65 : Channel 157
>> > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D
>> > tcp_tx(Without threaded NAPI)
>> > 317.34+-2.37 : Channel 44
>> > 317.01+-2.56 : Channel 157
>> >
>> > tcp_tx(With threaded NAPI)
>> > 371.34+-2.36 : Channel 44
>> > 376.95+-9.40 : Channel 157
>>
>> My concern is primarily with the induced tcp latency on this test.= A
>> way to check that is to run wireshark on your test client driving = the
>> test, capture the iperf traffic, and then plot the "Statistic= s->TCP
>> stream statistics for both throughput and rtt. Would it be possibl= e
>> for you to do that and put up those plots somewhere?
>>
>> The worst case test is a tcp bidirectional test which I don't = know if
>> older iperfs can do. (iperf2 has new bounceback and bidir tests) >>
>> Ideally stuff going in either direction, would not look as horribl= e,
>> as it did, back in 2016, documented in this linuxplumbers presenta= tion
>> here: https://blog.linuxplumbersconf.org/2016/ocw/sy= stem/presentations/3963/original/linuxplumbers_wifi_latency-3Nov.pdf >> and discussed on lwn, here: https://lwn.net/Articles/705884= /
>>
>> I worry about folk achieving slightly better tcp throughput at the=
>> expense of clobbering in-tcp-stream latency. Back then we were
>> shooting for no more than 40ms extra latency under load on this ch= ip,
>> down from (unusable) seconds. Presently elsewhere, on other chips,=
>> we're getting 8ms with stuff that's not in tree for the at= h10k, there
>> is a slight cost in single stream throughput but when multiple str= eams
>> are in use, on multiple stations, things like web pages fly,
>> irrespective of load.
>>
>>
>> > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D
>> >
>> > >
>> > > On Tue, Dec 20, 2022 at 12:17 AM Abhishek Kumar <kuabhs@chromium.org> wrote:
>> > > >
>> > > > NAPI poll can be done in threaded context along wit= h soft irq
>> > > > context. Threaded context can be scheduled efficien= tly, thus
>> > > > creating less of bottleneck during Rx processing. T= his patch is
>> > > > to enable threaded NAPI on ath10k driver.
>> > > >
>> > > > Based on testing, it was observed that on WCN3990, = the CPU0 reaches
>> > > > 100% utilization when napi runs in softirq context.= At the same
>> > > > time the other CPUs are at low consumption percenta= ge. This
>> > > > does not allow device to reach its maximum throughp= ut potential.
>> > > > After enabling threaded napi, CPU load is balanced = across all CPUs
>> > > > and following improvments were observed:
>> > > > - UDP_RX increase by ~22-25%
>> > > > - TCP_RX increase by ~15%
>> > > >
>> > > > Tested-on: WCN3990 hw1.0 SNOC WLAN.HL.3.2.2-00696-Q= CAHLSWMTPL-1
>> > > > Signed-off-by: Abhishek Kumar <kuabhs@chromium.org>
>> > > > ---
>> > > >
>> > > >=C2=A0 drivers/net/wireless/ath/ath10k/core.c | 16 += +++++++++++++++
>> > > >=C2=A0 drivers/net/wireless/ath/ath10k/hw.h=C2=A0 = =C2=A0|=C2=A0 2 ++
>> > > >=C2=A0 drivers/net/wireless/ath/ath10k/snoc.c |=C2= =A0 3 +++
>> > > >=C2=A0 3 files changed, 21 insertions(+)
>> > > >
>> > > > diff --git a/drivers/net/wireless/ath/ath10k/core.c= b/drivers/net/wireless/ath/ath10k/core.c
>> > > > index 5eb131ab916fd..ee4b6ba508c81 100644
>> > > > --- a/drivers/net/wireless/ath/ath10k/core.c
>> > > > +++ b/drivers/net/wireless/ath/ath10k/core.c
>> > > > @@ -100,6 +100,7 @@ static const struct ath10k_hw_p= arams ath10k_hw_params_list[] =3D {
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0.hw_restart_disconnect =3D false,
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0.use_fw_tx_credits =3D true,
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0.delay_unmap_buffer =3D false,
>> > > > +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0.enable_threaded_napi =3D false,
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0},
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0{
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0.id =3D QCA988X_HW_2_0_VERSION,
>> > > > @@ -140,6 +141,7 @@ static const struct ath10k_hw_p= arams ath10k_hw_params_list[] =3D {
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0.hw_restart_disconnect =3D false,
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0.use_fw_tx_credits =3D true,
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0.delay_unmap_buffer =3D false,
>> > > > +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0.enable_threaded_napi =3D false,
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0},
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0{
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0.id =3D QCA9887_HW_1_0_VERSION,
>> > > > @@ -181,6 +183,7 @@ static const struct ath10k_hw_p= arams ath10k_hw_params_list[] =3D {
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0.hw_restart_disconnect =3D false,
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0.use_fw_tx_credits =3D true,
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0.delay_unmap_buffer =3D false,
>> > > > +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0.enable_threaded_napi =3D false,
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0},
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0{
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0.id =3D QCA6174_HW_3_2_VERSION,
>> > > > @@ -217,6 +220,7 @@ static const struct ath10k_hw_p= arams ath10k_hw_params_list[] =3D {
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0.hw_restart_disconnect =3D false,
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0.use_fw_tx_credits =3D true,
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0.delay_unmap_buffer =3D false,
>> > > > +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0.enable_threaded_napi =3D false,
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0},
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0{
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0.id =3D QCA6174_HW_2_1_VERSION,
>> > > > @@ -257,6 +261,7 @@ static const struct ath10k_hw_p= arams ath10k_hw_params_list[] =3D {
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0.hw_restart_disconnect =3D false,
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0.use_fw_tx_credits =3D true,
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0.delay_unmap_buffer =3D false,
>> > > > +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0.enable_threaded_napi =3D false,
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0},
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0{
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0.id =3D QCA6174_HW_2_1_VERSION,
>> > > > @@ -297,6 +302,7 @@ static const struct ath10k_hw_p= arams ath10k_hw_params_list[] =3D {
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0.hw_restart_disconnect =3D false,
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0.use_fw_tx_credits =3D true,
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0.delay_unmap_buffer =3D false,
>> > > > +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0.enable_threaded_napi =3D false,
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0},
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0{
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0.id =3D QCA6174_HW_3_0_VERSION,
>> > > > @@ -337,6 +343,7 @@ static const struct ath10k_hw_p= arams ath10k_hw_params_list[] =3D {
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0.hw_restart_disconnect =3D false,
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0.use_fw_tx_credits =3D true,
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0.delay_unmap_buffer =3D false,
>> > > > +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0.enable_threaded_napi =3D false,
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0},
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0{
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0.id =3D QCA6174_HW_3_2_VERSION,
>> > > > @@ -381,6 +388,7 @@ static const struct ath10k_hw_p= arams ath10k_hw_params_list[] =3D {
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0.hw_restart_disconnect =3D false,
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0.use_fw_tx_credits =3D true,
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0.delay_unmap_buffer =3D false,
>> > > > +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0.enable_threaded_napi =3D false,
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0},
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0{
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0.id =3D QCA99X0_HW_2_0_DEV_VERSION,
>> > > > @@ -427,6 +435,7 @@ static const struct ath10k_hw_p= arams ath10k_hw_params_list[] =3D {
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0.hw_restart_disconnect =3D false,
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0.use_fw_tx_credits =3D true,
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0.delay_unmap_buffer =3D false,
>> > > > +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0.enable_threaded_napi =3D false,
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0},
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0{
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0.id =3D QCA9984_HW_1_0_DEV_VERSION,
>> > > > @@ -480,6 +489,7 @@ static const struct ath10k_hw_p= arams ath10k_hw_params_list[] =3D {
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0.hw_restart_disconnect =3D false,
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0.use_fw_tx_credits =3D true,
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0.delay_unmap_buffer =3D false,
>> > > > +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0.enable_threaded_napi =3D false,
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0},
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0{
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0.id =3D QCA9888_HW_2_0_DEV_VERSION,
>> > > > @@ -530,6 +540,7 @@ static const struct ath10k_hw_p= arams ath10k_hw_params_list[] =3D {
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0.hw_restart_disconnect =3D false,
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0.use_fw_tx_credits =3D true,
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0.delay_unmap_buffer =3D false,
>> > > > +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0.enable_threaded_napi =3D false,
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0},
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0{
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0.id =3D QCA9377_HW_1_0_DEV_VERSION,
>> > > > @@ -570,6 +581,7 @@ static const struct ath10k_hw_p= arams ath10k_hw_params_list[] =3D {
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0.hw_restart_disconnect =3D false,
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0.use_fw_tx_credits =3D true,
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0.delay_unmap_buffer =3D false,
>> > > > +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0.enable_threaded_napi =3D false,
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0},
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0{
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0.id =3D QCA9377_HW_1_1_DEV_VERSION,
>> > > > @@ -612,6 +624,7 @@ static const struct ath10k_hw_p= arams ath10k_hw_params_list[] =3D {
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0.hw_restart_disconnect =3D false,
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0.use_fw_tx_credits =3D true,
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0.delay_unmap_buffer =3D false,
>> > > > +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0.enable_threaded_napi =3D false,
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0},
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0{
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0.id =3D QCA9377_HW_1_1_DEV_VERSION,
>> > > > @@ -645,6 +658,7 @@ static const struct ath10k_hw_p= arams ath10k_hw_params_list[] =3D {
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0.hw_restart_disconnect =3D false,
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0.use_fw_tx_credits =3D true,
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0.delay_unmap_buffer =3D false,
>> > > > +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0.enable_threaded_napi =3D false,
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0},
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0{
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0.id =3D QCA4019_HW_1_0_DEV_VERSION,
>> > > > @@ -692,6 +706,7 @@ static const struct ath10k_hw_p= arams ath10k_hw_params_list[] =3D {
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0.hw_restart_disconnect =3D false,
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0.use_fw_tx_credits =3D true,
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0.delay_unmap_buffer =3D false,
>> > > > +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0.enable_threaded_napi =3D false,
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0},
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0{
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0.id =3D WCN3990_HW_1_0_DEV_VERSION,
>> > > > @@ -725,6 +740,7 @@ static const struct ath10k_hw_p= arams ath10k_hw_params_list[] =3D {
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0.hw_restart_disconnect =3D true,
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0.use_fw_tx_credits =3D false,
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0.delay_unmap_buffer =3D true,
>> > > > +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0.enable_threaded_napi =3D true,
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0},
>> > > >=C2=A0 };
>> > > >
>> > > > diff --git a/drivers/net/wireless/ath/ath10k/hw.h b= /drivers/net/wireless/ath/ath10k/hw.h
>> > > > index 9643031a4427a..adf3076b96503 100644
>> > > > --- a/drivers/net/wireless/ath/ath10k/hw.h
>> > > > +++ b/drivers/net/wireless/ath/ath10k/hw.h
>> > > > @@ -639,6 +639,8 @@ struct ath10k_hw_params {
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0bool use_fw_tx_cre= dits;
>> > > >
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0bool delay_unmap_b= uffer;
>> > > > +
>> > > > +=C2=A0 =C2=A0 =C2=A0 =C2=A0bool enable_threaded_na= pi;
>> > > >=C2=A0 };
>> > > >
>> > > >=C2=A0 struct htt_resp;
>> > > > diff --git a/drivers/net/wireless/ath/ath10k/snoc.c= b/drivers/net/wireless/ath/ath10k/snoc.c
>> > > > index cfcb759a87dea..b94150fb6ef06 100644
>> > > > --- a/drivers/net/wireless/ath/ath10k/snoc.c
>> > > > +++ b/drivers/net/wireless/ath/ath10k/snoc.c
>> > > > @@ -927,6 +927,9 @@ static int ath10k_snoc_hif_star= t(struct ath10k *ar)
>> > > >
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0bitmap_clear(ar_sn= oc->pending_ce_irqs, 0, CE_COUNT_MAX);
>> > > >
>> > > > +=C2=A0 =C2=A0 =C2=A0 =C2=A0if (ar->hw_params.en= able_threaded_napi)
>> > > > +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0dev_set_threaded(&ar->napi_dev, true);
>> > > > +
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0ath10k_core_napi_e= nable(ar);
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0ath10k_snoc_irq_en= able(ar);
>> > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0ath10k_snoc_rx_pos= t(ar);
>> > > > --
>> > > > 2.39.0.314.g84b9a713c41-goog
>> > > >
>> > >
>> > >
>> > > --
>> > > This song goes out to all the folk that thought Stadia w= ould work:
>> > > https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981= 366665607352320-FXtz
>> > > Dave T=C3=A4ht CEO, TekLibre, LLC
>>
>>
>>
>> --
>> This song goes out to all the folk that thought Stadia would work:=
>> htt= ps://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-69813666656073= 52320-FXtz
>> Dave T=C3=A4ht CEO, TekLibre, LLC
>> _______________________________________________
>> Make-wifi-fast mailing list
>> Make-wifi-fast@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo= /make-wifi-fast
>
>
> This electronic communication and the information and any files transm= itted with it, or attached to it, are confidential and are intended solely = for the use of the individual or entity to whom it is addressed and may con= tain information that is confidential, legally privileged, protected by pri= vacy laws, or otherwise restricted from disclosure to anyone else. If you a= re not the intended recipient or the person responsible for delivering the = e-mail to the intended recipient, you are hereby notified that any use, cop= ying, distributing, dissemination, forwarding, printing, or copying of this= e-mail is strictly prohibited. If you received this e-mail in error, pleas= e return the e-mail to the sender, delete it from your computer, and destro= y any printed copy of it.



--
This song goes out to all the folk that thought Stadia would work:
https://www.= linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXt= z
Dave T=C3=A4ht CEO, TekLibre, LLC

This ele= ctronic communication and the information and any files transmitted with it= , or attached to it, are confidential and are intended solely for the use o= f the individual or entity to whom it is addressed and may contain informat= ion that is confidential, legally privileged, protected by privacy laws, or= otherwise restricted from disclosure to anyone else. If you are not the in= tended recipient or the person responsible for delivering the e-mail to the= intended recipient, you are hereby notified that any use, copying, distrib= uting, dissemination, forwarding, printing, or copying of this e-mail is st= rictly prohibited. If you received this e-mail in error, please return the = e-mail to the sender, delete it from your computer, and destroy any printed= copy of it. --00000000000070b9d205f1119f52-- --00000000000078979705f1119f1c Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIIQagYJKoZIhvcNAQcCoIIQWzCCEFcCAQExDzANBglghkgBZQMEAgEFADALBgkqhkiG9w0BBwGg gg3BMIIFDTCCA/WgAwIBAgIQeEqpED+lv77edQixNJMdADANBgkqhkiG9w0BAQsFADBMMSAwHgYD VQQLExdHbG9iYWxTaWduIFJvb3QgQ0EgLSBSMzETMBEGA1UEChMKR2xvYmFsU2lnbjETMBEGA1UE AxMKR2xvYmFsU2lnbjAeFw0yMDA5MTYwMDAwMDBaFw0yODA5MTYwMDAwMDBaMFsxCzAJBgNVBAYT AkJFMRkwFwYDVQQKExBHbG9iYWxTaWduIG52LXNhMTEwLwYDVQQDEyhHbG9iYWxTaWduIEdDQyBS MyBQZXJzb25hbFNpZ24gMiBDQSAyMDIwMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA vbCmXCcsbZ/a0fRIQMBxp4gJnnyeneFYpEtNydrZZ+GeKSMdHiDgXD1UnRSIudKo+moQ6YlCOu4t rVWO/EiXfYnK7zeop26ry1RpKtogB7/O115zultAz64ydQYLe+a1e/czkALg3sgTcOOcFZTXk38e aqsXsipoX1vsNurqPtnC27TWsA7pk4uKXscFjkeUE8JZu9BDKaswZygxBOPBQBwrA5+20Wxlk6k1 e6EKaaNaNZUy30q3ArEf30ZDpXyfCtiXnupjSK8WU2cK4qsEtj09JS4+mhi0CTCrCnXAzum3tgcH cHRg0prcSzzEUDQWoFxyuqwiwhHu3sPQNmFOMwIDAQABo4IB2jCCAdYwDgYDVR0PAQH/BAQDAgGG MGAGA1UdJQRZMFcGCCsGAQUFBwMCBggrBgEFBQcDBAYKKwYBBAGCNxQCAgYKKwYBBAGCNwoDBAYJ KwYBBAGCNxUGBgorBgEEAYI3CgMMBggrBgEFBQcDBwYIKwYBBQUHAxEwEgYDVR0TAQH/BAgwBgEB /wIBADAdBgNVHQ4EFgQUljPR5lgXWzR1ioFWZNW+SN6hj88wHwYDVR0jBBgwFoAUj/BLf6guRSSu TVD6Y5qL3uLdG7wwegYIKwYBBQUHAQEEbjBsMC0GCCsGAQUFBzABhiFodHRwOi8vb2NzcC5nbG9i YWxzaWduLmNvbS9yb290cjMwOwYIKwYBBQUHMAKGL2h0dHA6Ly9zZWN1cmUuZ2xvYmFsc2lnbi5j b20vY2FjZXJ0L3Jvb3QtcjMuY3J0MDYGA1UdHwQvMC0wK6ApoCeGJWh0dHA6Ly9jcmwuZ2xvYmFs c2lnbi5jb20vcm9vdC1yMy5jcmwwWgYDVR0gBFMwUTALBgkrBgEEAaAyASgwQgYKKwYBBAGgMgEo CjA0MDIGCCsGAQUFBwIBFiZodHRwczovL3d3dy5nbG9iYWxzaWduLmNvbS9yZXBvc2l0b3J5LzAN BgkqhkiG9w0BAQsFAAOCAQEAdAXk/XCnDeAOd9nNEUvWPxblOQ/5o/q6OIeTYvoEvUUi2qHUOtbf jBGdTptFsXXe4RgjVF9b6DuizgYfy+cILmvi5hfk3Iq8MAZsgtW+A/otQsJvK2wRatLE61RbzkX8 9/OXEZ1zT7t/q2RiJqzpvV8NChxIj+P7WTtepPm9AIj0Keue+gS2qvzAZAY34ZZeRHgA7g5O4TPJ /oTd+4rgiU++wLDlcZYd/slFkaT3xg4qWDepEMjT4T1qFOQIL+ijUArYS4owpPg9NISTKa1qqKWJ jFoyms0d0GwOniIIbBvhI2MJ7BSY9MYtWVT5jJO3tsVHwj4cp92CSFuGwunFMzCCA18wggJHoAMC AQICCwQAAAAAASFYUwiiMA0GCSqGSIb3DQEBCwUAMEwxIDAeBgNVBAsTF0dsb2JhbFNpZ24gUm9v dCBDQSAtIFIzMRMwEQYDVQQKEwpHbG9iYWxTaWduMRMwEQYDVQQDEwpHbG9iYWxTaWduMB4XDTA5 MDMxODEwMDAwMFoXDTI5MDMxODEwMDAwMFowTDEgMB4GA1UECxMXR2xvYmFsU2lnbiBSb290IENB IC0gUjMxEzARBgNVBAoTCkdsb2JhbFNpZ24xEzARBgNVBAMTCkdsb2JhbFNpZ24wggEiMA0GCSqG SIb3DQEBAQUAA4IBDwAwggEKAoIBAQDMJXaQeQZ4Ihb1wIO2hMoonv0FdhHFrYhy/EYCQ8eyip0E XyTLLkvhYIJG4VKrDIFHcGzdZNHr9SyjD4I9DCuul9e2FIYQebs7E4B3jAjhSdJqYi8fXvqWaN+J J5U4nwbXPsnLJlkNc96wyOkmDoMVxu9bi9IEYMpJpij2aTv2y8gokeWdimFXN6x0FNx04Druci8u nPvQu7/1PQDhBjPogiuuU6Y6FnOM3UEOIDrAtKeh6bJPkC4yYOlXy7kEkmho5TgmYHWyn3f/kRTv riBJ/K1AFUjRAjFhGV64l++td7dkmnq/X8ET75ti+w1s4FRpFqkD2m7pg5NxdsZphYIXAgMBAAGj QjBAMA4GA1UdDwEB/wQEAwIBBjAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBSP8Et/qC5FJK5N UPpjmove4t0bvDANBgkqhkiG9w0BAQsFAAOCAQEAS0DbwFCq/sgM7/eWVEVJu5YACUGssxOGhigH M8pr5nS5ugAtrqQK0/Xx8Q+Kv3NnSoPHRHt44K9ubG8DKY4zOUXDjuS5V2yq/BKW7FPGLeQkbLmU Y/vcU2hnVj6DuM81IcPJaP7O2sJTqsyQiunwXUaMld16WCgaLx3ezQA3QY/tRG3XUyiXfvNnBB4V 14qWtNPeTCekTBtzc3b0F5nCH3oO4y0IrQocLP88q1UOD5F+NuvDV0m+4S4tfGCLw0FREyOdzvcy a5QBqJnnLDMfOjsl0oZAzjsshnjJYS8Uuu7bVW/fhO4FCU29KNhyztNiUGUe65KXgzHZs7XKR1g/ XzCCBUkwggQxoAMCAQICDDGs4Qlq5OZK9mcDzTANBgkqhkiG9w0BAQsFADBbMQswCQYDVQQGEwJC RTEZMBcGA1UEChMQR2xvYmFsU2lnbiBudi1zYTExMC8GA1UEAxMoR2xvYmFsU2lnbiBHQ0MgUjMg UGVyc29uYWxTaWduIDIgQ0EgMjAyMDAeFw0yMjA5MTAxMzMzNDFaFw0yNTA5MTAxMzMzNDFaMIGM MQswCQYDVQQGEwJJTjESMBAGA1UECBMJS2FybmF0YWthMRIwEAYDVQQHEwlCYW5nYWxvcmUxFjAU BgNVBAoTDUJyb2FkY29tIEluYy4xFDASBgNVBAMTC0JvYiBNY01haG9uMScwJQYJKoZIhvcNAQkB Fhhib2IubWNtYWhvbkBicm9hZGNvbS5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIB AQDBfX3nsBFRdO26im8lhOadVadRmV/YWK+U9OoGlTE+2MDsjJwO5p/Q6iaTUropqMRH1E+EIuhe /OU6a3/btrqzARE77RaVSdz5swXt7M4ciN+z44nIEx36UQIlFLsBFa3is/J/QLFhTUFFf0wLJsUO wyja+KvygH/E5TyfeXf5T2Y2wjGZx8jQXZMDmNpfANlEBYDfzCNYcAIQNox8FuPpEpuxWvv7jvxV X5dfkSef9T/DbsDM0PeTVMVyYIQoRSMBIGxVkaqp0MJglvQ2mU4CXcoOGgm6XC8LoLoEvYojXFKC fRgCOT5xeMR10UPSBQIljKwt7fPhpYVY+jTtOclpAgMBAAGjggHZMIIB1TAOBgNVHQ8BAf8EBAMC BaAwgaMGCCsGAQUFBwEBBIGWMIGTME4GCCsGAQUFBzAChkJodHRwOi8vc2VjdXJlLmdsb2JhbHNp Z24uY29tL2NhY2VydC9nc2djY3IzcGVyc29uYWxzaWduMmNhMjAyMC5jcnQwQQYIKwYBBQUHMAGG NWh0dHA6Ly9vY3NwLmdsb2JhbHNpZ24uY29tL2dzZ2NjcjNwZXJzb25hbHNpZ24yY2EyMDIwME0G A1UdIARGMEQwQgYKKwYBBAGgMgEoCjA0MDIGCCsGAQUFBwIBFiZodHRwczovL3d3dy5nbG9iYWxz aWduLmNvbS9yZXBvc2l0b3J5LzAJBgNVHRMEAjAAMEkGA1UdHwRCMEAwPqA8oDqGOGh0dHA6Ly9j cmwuZ2xvYmFsc2lnbi5jb20vZ3NnY2NyM3BlcnNvbmFsc2lnbjJjYTIwMjAuY3JsMCMGA1UdEQQc MBqBGGJvYi5tY21haG9uQGJyb2FkY29tLmNvbTATBgNVHSUEDDAKBggrBgEFBQcDBDAfBgNVHSME GDAWgBSWM9HmWBdbNHWKgVZk1b5I3qGPzzAdBgNVHQ4EFgQUpG/4RP1YQA/iXGens9pIRe7CQxMw DQYJKoZIhvcNAQELBQADggEBACfWLy4qJyCnOa3sl4LEDAMU/gmJ6LbclGE5iR4KanAmlAt92gzN 5lSy/iE+wsRrXiHI7YKFgXX1kVK/RqMiPRrw4hq2j8nxoSi/VFiyS3CsfVMGkbY7HBTlBvla/tH+ +2nJprlXbJyz1GdvoJAeam5RvTWotcCGAjZmMa3U3zMkszgXN849xe3dUK1DauUGiInXEwEdXDcA /0CVjL3EEMj+kNWcLhrSZKwFtxggUyMW3XWRaAeAL9wOtEaXYqlgbtnV0n9FuoV2TNm3h7Mh7rjV I2zM+IZ3DE+XFK7dcPwte33u75QyySNJ3UMZqi25CO85yl8Bmo7aWRm99N7HGnkxggJtMIICaQIB ATBrMFsxCzAJBgNVBAYTAkJFMRkwFwYDVQQKExBHbG9iYWxTaWduIG52LXNhMTEwLwYDVQQDEyhH bG9iYWxTaWduIEdDQyBSMyBQZXJzb25hbFNpZ24gMiBDQSAyMDIwAgwxrOEJauTmSvZnA80wDQYJ YIZIAWUDBAIBBQCggdQwLwYJKoZIhvcNAQkEMSIEIAv1sHCcA1YURVkSNDnHmnWjn/aLWm4lp0Y7 J/gxAe1nMBgGCSqGSIb3DQEJAzELBgkqhkiG9w0BBwEwHAYJKoZIhvcNAQkFMQ8XDTIyMTIzMDIw NDExNVowaQYJKoZIhvcNAQkPMVwwWjALBglghkgBZQMEASowCwYJYIZIAWUDBAEWMAsGCWCGSAFl AwQBAjAKBggqhkiG9w0DBzALBgkqhkiG9w0BAQowCwYJKoZIhvcNAQEHMAsGCWCGSAFlAwQCATAN BgkqhkiG9w0BAQEFAASCAQAr34+2c1oDBo1e+07pcnhHBGitSxCD+0k9/dI4V4JMPgBMzKj7FBg4 osyfEUmua8YM5ZrG81DNcoR24hyiO+eb1YHF5EJbiWuKFV13xjPPq0ci40DwECCSROGVndrIH0rD OGZTB01ExdRHe7oxoyo23YMi2UMzg4uRxM324UdtG68qsE7i6+mFPDld5csOmJeQhO5ka9h4yI9F o1NwTyVXog4ZB6IPgGLjVHqToNh4wEcGqLs8XxkhmA0UzXWLMRgRoxORJTwOzYMN1z/F+umF0BDl 8USZubxwTaOsaI+1gUZzizhAVBqtPfObc7bak5Wu0pvyRcLYrO8TLcmrPsxb --00000000000078979705f1119f1c--