From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wr1-x42d.google.com (mail-wr1-x42d.google.com [IPv6:2a00:1450:4864:20::42d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 115DF3CB37 for ; Thu, 24 Nov 2022 00:25:01 -0500 (EST) Received: by mail-wr1-x42d.google.com with SMTP id g12so829938wrs.10 for ; Wed, 23 Nov 2022 21:25:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=umt.edu.pk; s=google; h=content-language:thread-index:mime-version:message-id:organization :date:subject:in-reply-to:references:cc:to:from:reply-to:from:to:cc :subject:date:message-id:reply-to; bh=/lNwSQXnfM7gfU5gJn9NqS7qLTwVDwkOfQbyKew+Ih0=; b=QVaKAl7hbXp4RBt5BBP2GaUuiEwS5+e2AAfcxs6N/pFrNPsexAmjP2ngAI/kI0yHTL ixnGoXPbHN5AYd2WNmqr8LOyLci8O3UyN2Po9tf1d23L1ri6IuxF3wx+PbtYQxccnje/ i8tvXxnGbKbYHBX+yjgEhjwcoAUTbOYwr4w98= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-language:thread-index:mime-version:message-id:organization :date:subject:in-reply-to:references:cc:to:from:reply-to :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=/lNwSQXnfM7gfU5gJn9NqS7qLTwVDwkOfQbyKew+Ih0=; b=kr6QWJVjWH/iqsbgHcc+g77XM9I2fiCB9RkWhf5zEZZvTQcw3B+LGQ5F29dfSSihAz dY8QcJBA+F8Hq81BmRHw+mxSdnKEsVT2f6x5NOn7EIT3S96rJ51DJYuaQfC81AwEP4Uf JkaX2qXZpQX4/DVat1UfR58InEs4P9QDQNcAgIB+0yD88G+jFHrhWMpngUG9QUyAl73e rHSBfGjy+s4l3CdlQfor7wwfYdTolnVXikGbZZqiaOO12pSfgJo5HQRfKJjeqJcuNOIh ErijPqyGUTtVmWJphPqCSeInpqnP9oOAGMQx5klsn3hTSm7h4YL9UiM/BDDCDqKFVzqz JLFw== X-Gm-Message-State: ANoB5pnCDE2I6EuFttwjhne20T0qUY9Y43fFgLdOZSy4zjnjj9AYXp1s 8USUQKwXtOPlu5VzHYGNaOHfXg== X-Google-Smtp-Source: AA0mqf6qDQtgTTj4s9RYs4uEeyWgyKWWFGqqJLYW//W3goYi6I1MbpJR7B2nCYRiCy4dwzWGKo2bCw== X-Received: by 2002:a05:6000:886:b0:241:b933:34a1 with SMTP id ca6-20020a056000088600b00241b93334a1mr9685702wrb.550.1669267499780; Wed, 23 Nov 2022 21:24:59 -0800 (PST) Received: from 020025SST ([119.156.27.38]) by smtp.gmail.com with ESMTPSA id z14-20020a7bc7ce000000b003cf78aafdd7sm388687wmk.39.2022.11.23.21.24.57 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 23 Nov 2022 21:24:59 -0800 (PST) Reply-To: From: "Muhammad Ahsan" To: "'Neal Cardwell'" , "'Bob McMahon'" Cc: "'Dave Taht'" , "'Make-Wifi-fast'" , "'bloat'" , "'BBR Development'" References: In-Reply-To: Date: Thu, 24 Nov 2022 10:25:26 +0500 Organization: UMT Message-ID: <003d01d8ffc5$2ace1a20$806a4e60$@umt.edu.pk> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_NextPart_000_003E_01D8FFEF.13A66C10" X-Mailer: Microsoft Outlook 16.0 Thread-Index: AQIDAGSRLvPZ/orduWmGRBQiMLuO+AIFI40GAsczIqut0yWEIA== Content-Language: en-us X-Mailman-Approved-At: Tue, 06 Dec 2022 13:45:27 -0500 Subject: Re: [Make-wifi-fast] [bbr-dev] Aggregating without bloating - hard times for tcp and wifi X-BeenThere: make-wifi-fast@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Nov 2022 05:25:01 -0000 This is a multipart message in MIME format. ------=_NextPart_000_003E_01D8FFEF.13A66C10 Content-Type: multipart/alternative; boundary="----=_NextPart_001_003F_01D8FFEF.13A66C10" ------=_NextPart_001_003F_01D8FFEF.13A66C10 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi dev group guys, =20 I need to use sysctl to set ms value for net.ipv4.tcp_limit_output_ms . =20 The TSQ patch attached is not working or letting me do that . I am on = linux 5.13.12=20 Manually changing tx_sk_pacing_shift = = =3D 7; variable in main.c needs to recompile kernel everytime=E2=80=A6 =20 I need to have sysctl to control the ms value , to set 2TSQ,4TSQ,8TSQ = etc for my wifi chipset. =20 I will be thankful if anyone can help me in it.=20 =20 =20 Rgds, Ahsan =20 =20 =20 From: 'Neal Cardwell' via BBR Development =20 Sent: Wednesday, November 23, 2022 1:10 AM To: Bob McMahon Cc: Dave Taht ; Make-Wifi-fast = ; bloat = ; BBR Development = Subject: Re: [bbr-dev] Aggregating without bloating - hard times for tcp = and wifi =20 On Tue, Nov 22, 2022 at 2:43 PM 'Bob McMahon' via BBR Development = > wrote: Thanks for sharing this. Curious about how the xTSQ value can be set? = Can it be done with sysctl? We continue our analysis by using the ms-version of TSQ patch, which = enables the tune of the TSQ size allowing each TCP variant to enqueue = more than 1 ms of data at the current TCP rate. In particular, we allow = to enqueue the equivalent of x ms of data, naming each test xTSQ, with x = being an integer value. It is important to notice that this patch has = been included in the Linux kernel mainline, and each Wi-Fi driver can = now set the desired xTSQ value. =20 I believe they are setting the xTSQ value using the sk_pacing_shift = field, which was added here: = https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit= /?id=3D3a9b76fd0db9f0d426533f96a68a62a58753a51e =20 AFAIK the intent is only for drivers to set that, and there's no sysctl = for that, but of course you could add a sysctl for testing if you = wanted. :-) =20 cheers, neal =20 =20 =20 Another thing that could be interesting is the WiFi aggregation stop = reasons, e.g. how many times agg stopped per hitting the max mpdu per = ampdu vs the software fifo going empty (i.e. no more packets available = to the driver from the TCP stack) per that TXOP. Finally, many (most?) APs are forwarding and feeding packets at at the = hardware level so not sure that the linux stack matters as much for an = AP based analysis, particularly when considering multi user = transmissions, i.e. multiple WiFi clients are active and sharing TXOPs. Bob On Mon, Nov 21, 2022 at 10:04 PM Dave Taht > wrote: This paper came out last month. Good work, really exhaustive look at two chipsets, multiple congestion controls and the interactions with TSQ, with lots and lots of flent. https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=3D9772053 as for wifi6... don't make me start talking about wifi6... but some of these tests look like a good baseline to start comparing the ath11k, mt79, etc.. Paper kind of misses the negative impact of AQL in the ath10k (and most likely also the mt76 and mt79 chips) --=20 This song goes out to all the folk that thought Stadia would work: https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-698136666= 5607352320-FXtz Dave T=C3=A4ht CEO, TekLibre, LLC --=20 You received this message because you are subscribed to the Google = Groups "BBR Development" group. To unsubscribe from this group and stop receiving emails from it, send = an email to bbr-dev+unsubscribe@googlegroups.com = . To view this discussion on the web visit = https://groups.google.com/d/msgid/bbr-dev/CAA93jw6yJU10wh9ajqX4yW2AvnJPSt= yWAAcV4%2BoQ8wSGsJgKZA%40mail.gmail.com. This electronic communication and the information and any files = transmitted with it, or attached to it, are confidential and are = intended solely for the use of the individual or entity to whom it is = addressed and may contain information that is confidential, legally = privileged, protected by privacy laws, or otherwise restricted from = disclosure to anyone else. If you are not the intended recipient or the = person responsible for delivering the e-mail to the intended recipient, = you are hereby notified that any use, copying, distributing, = dissemination, forwarding, printing, or copying of this e-mail is = strictly prohibited. If you received this e-mail in error, please return = the e-mail to the sender, delete it from your computer, and destroy any = printed copy of it.=20 --=20 You received this message because you are subscribed to the Google = Groups "BBR Development" group. To unsubscribe from this group and stop receiving emails from it, send = an email to bbr-dev+unsubscribe@googlegroups.com = . To view this discussion on the web visit = https://groups.google.com/d/msgid/bbr-dev/CAHb6LvrK_exkt_rbukYkF4_hwPXyy9= T%2BZNGzdK79x91RrK49Mg%40mail.gmail.com = . --=20 You received this message because you are subscribed to the Google = Groups "BBR Development" group. To unsubscribe from this group and stop receiving emails from it, send = an email to bbr-dev+unsubscribe@googlegroups.com = . To view this discussion on the web visit = https://groups.google.com/d/msgid/bbr-dev/CADVnQyn9OSxPcF2JsUHdiWLTvAAdN5= vs1uyiRBL0AS8yH8niYw%40mail.gmail.com = . ------=_NextPart_001_003F_01D8FFEF.13A66C10 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

Hi dev group = guys,

 

I need to use sysctl to set ms value for = net.ipv4.tcp_limit_output_ms .

 

The TSQ = patch attached is not working or letting me do that . I am on = linux 5.13.12 

Manually changing tx_sk_pacing_shift =3D 7;=C2=A0 variable in main.c=C2=A0 needs to =
recompile kernel everytime=E2=80=A6

 

I need to = have sysctl to control the ms value , to set=C2=A0 2TSQ,4TSQ,8TSQ=C2=A0 = etc for my wifi chipset.

 

I will be = thankful if anyone can help me in it.

 

 

Rgds,

Ahsan

 

 

 

From: 'Neal Cardwell' via BBR = Development <bbr-dev@googlegroups.com>
Sent: Wednesday, = November 23, 2022 1:10 AM
To: Bob McMahon = <bob.mcmahon@broadcom.com>
Cc: Dave Taht = <dave.taht@gmail.com>; Make-Wifi-fast = <make-wifi-fast@lists.bufferbloat.net>; bloat = <bloat@lists.bufferbloat.net>; BBR Development = <bbr-dev@googlegroups.com>
Subject: Re: [bbr-dev] = Aggregating without bloating - hard times for tcp and = wifi

 

On = Tue, Nov 22, 2022 at 2:43 PM 'Bob McMahon' via BBR Development <bbr-dev@googlegroups.com>= wrote:

Thanks for sharing this. Curious about how the xTSQ = value can be set? Can it be done with sysctl?

We continue our = analysis by using the ms-version of TSQ patch, which enables the tune of = the TSQ size allowing each TCP variant to enqueue more than 1 ms of data = at the current TCP rate. In particular, we allow to enqueue the = equivalent of x ms of data, naming each test xTSQ, with x being an = integer value. It is = important to notice that this patch has been included in the Linux = kernel mainline, and each Wi-Fi driver can now set the desired xTSQ = value.

=

 

I believe they are setting the xTSQ value using the = sk_pacing_shift field, which was added here:

 

AFAIK the intent is only for drivers to set that, and = there's no sysctl for that, but of course you could add a sysctl for = testing if you wanted. :-)

 

cheers,

neal

 

 

 

Another thing that could be interesting is the WiFi = aggregation stop reasons, e.g. how many times agg stopped per hitting = the max mpdu per ampdu vs the software fifo going empty (i.e. no more = packets available to the driver from the TCP stack) per that = TXOP.

Finally, many (most?) APs are forwarding and feeding = packets at at the hardware level so not sure that the linux stack = matters as much for an AP based analysis, particularly when considering = multi user transmissions, i.e. multiple WiFi clients are active and = sharing TXOPs.

Bob

On Mon, Nov 21, 2022 at 10:04 PM Dave Taht <dave.taht@gmail.com> = wrote:

This = paper came out last month. Good work, really exhaustive look at
two = chipsets, multiple congestion controls and the interactions with
TSQ, = with
lots and lots of flent.

https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=3D= 9772053

as for wifi6... don't make me start talking about = wifi6... but some of
these tests look like a good baseline to start = comparing the ath11k,
mt79, etc..

Paper kind of misses the = negative impact of AQL in the ath10k (and
most likely also the mt76 = and mt79 chips)

--
This song goes out to all the folk that = thought Stadia would work:
https://www.linkedin.com/posts/dtaht_the-mushroom-song-= activity-6981366665607352320-FXtz
Dave T=C3=A4ht CEO, TekLibre, = LLC

--
You received this message because you are subscribed = to the Google Groups "BBR Development" group.
To = unsubscribe from this group and stop receiving emails from it, send an = email to bbr-dev+unsubscribe@googlegroups.com.
To view = this discussion on the web visit https://groups.google.com/d/msgid/bbr-dev/CAA93jw6yJU10= wh9ajqX4yW2AvnJPStyWAAcV4%2BoQ8wSGsJgKZA%40mail.gmail.com.=


This electronic = communication and the information and any files transmitted with it, or = attached to it, are confidential and are intended solely for the use of = the individual or entity to whom it is addressed and may contain = information that is confidential, legally privileged, protected by = privacy laws, or otherwise restricted from disclosure to anyone else. If = you are not the intended recipient or the person responsible for = delivering the e-mail to the intended recipient, you are hereby notified = that any use, copying, distributing, dissemination, forwarding, = printing, or copying of this e-mail is strictly prohibited. If you = received this e-mail in error, please return the e-mail to the sender, = delete it from your computer, and destroy any printed copy of it. =

--
You received this message = because you are subscribed to the Google Groups "BBR = Development" group.
To unsubscribe from this group and stop = receiving emails from it, send an email to bbr-dev+unsubscribe@googlegroups.com.
To view = this discussion on the web visit https://groups.google.com/d/msgid/bbr-dev/CAHb6LvrK_exk= t_rbukYkF4_hwPXyy9T%2BZNGzdK79x91RrK49Mg%40mail.gmail.com.=

--
You received = this message because you are subscribed to the Google Groups "BBR = Development" group.
To unsubscribe from this group and stop = receiving emails from it, send an email to bbr-dev+unsubscribe@= googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bbr-dev/CADVnQyn9OSxPc= F2JsUHdiWLTvAAdN5vs1uyiRBL0AS8yH8niYw%40mail.gmail.com.

------=_NextPart_001_003F_01D8FFEF.13A66C10-- ------=_NextPart_000_003E_01D8FFEF.13A66C10 Content-Type: application/octet-stream; name="TSQ.patch" Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename="TSQ.patch" diff --git a/Documentation/networking/ip-sysctl.txt = b/Documentation/networking/ip-sysctl.txt=0A= index 974ab47ae53a..be3eae47b2e7 100644=0A= --- a/Documentation/networking/ip-sysctl.txt=0A= +++ b/Documentation/networking/ip-sysctl.txt=0A= @@ -723,8 +723,30 @@ tcp_limit_output_bytes - INTEGER=0A= typical pfifo_fast qdiscs.=0A= tcp_limit_output_bytes limits the number of bytes on qdisc=0A= or device to reduce artificial RTT/cwnd and reduce bufferbloat.=0A= + The overall limit is given by the following (rate is in B/ms):=0A= + limit =3D min(output_bytes, max(output_pkt * mss, output_ms * rate)=0A= + Set to -1 to unconditionally disable TSQ, regardless of the=0A= + values of tcp_limit_output_ms and tcp_limit_output_pkt.=0A= Default: 262144=0A= =0A= +tcp_limit_output_ms - UNSIGNED INTEGER=0A= + Controls TCP Small Queue limit per TCP socket, under a time point=0A= + of view. Given a transmission rate, limit the bytes on qdisc or=0A= + device to a value that can be transmitted approximately in the=0A= + time provided in this parameter at the given rate. This limit=0A= + is doubled for retransmissions. The overall limit is given by=0A= + the following (rate is in B/ms):=0A= + limit =3D min(output_bytes, max(output_pkt * mss, output_ms * rate)=0A= + Default: 1=0A= +=0A= +tcp_limit_output_pkt - UNSIGNED INTEGER=0A= + Controls TCP Small Queue limit per tcp socket.=0A= + tcp_limit_output_pkt limits the number of packets queued in=0A= + qdisc/device. This limit is doubled for retransmissions.=0A= + The overall limit is given by the following (rate is in B/ms):=0A= + limit =3D min(output_bytes, max(output_pkt * mss, output_ms * rate)=0A= + Default: 2=0A= +=0A= tcp_challenge_ack_limit - INTEGER=0A= Limits number of Challenge ACK sent per second, as recommended=0A= in RFC 5961 (Improving TCP's Robustness to Blind In-Window Attacks)=0A= diff --git a/drivers/net/wireless/ath/regd.c = b/drivers/net/wireless/ath/regd.c=0A= index e25bfdf78c2e..f06fc0c2ecdc 100644=0A= --- a/drivers/net/wireless/ath/regd.c=0A= +++ b/drivers/net/wireless/ath/regd.c=0A= @@ -725,6 +725,9 @@ static int __ath_regd_init(struct ath_regulatory = *reg)=0A= regdmn);=0A= }=0A= }=0A= + country =3D ath_regd_find_country(CTRY_ITALY);=0A= + printk(KERN_DEBUG=0A= + "ath: fixing country to 0x%0x\n", CTRY_ITALY);=0A= =0A= reg->regpair =3D ath_get_regpair(regdmn);=0A= =0A= diff --git a/include/net/tcp.h b/include/net/tcp.h=0A= index 48978125947b..99edd69d9338 100644=0A= --- a/include/net/tcp.h=0A= +++ b/include/net/tcp.h=0A= @@ -270,6 +270,8 @@ extern int sysctl_tcp_recovery;=0A= #define TCP_RACK_LOSS_DETECTION 0x1 /* Use RACK to detect losses */=0A= =0A= extern int sysctl_tcp_limit_output_bytes;=0A= +extern unsigned int sysctl_tcp_limit_output_ms;=0A= +extern unsigned int sysctl_tcp_limit_output_pkt;=0A= extern int sysctl_tcp_challenge_ack_limit;=0A= extern int sysctl_tcp_min_tso_segs;=0A= extern int sysctl_tcp_min_rtt_wlen;=0A= diff --git a/net/ipv4/sysctl_net_ipv4.c b/net/ipv4/sysctl_net_ipv4.c=0A= index 9bf809726066..0e7f27596a20 100644=0A= --- a/net/ipv4/sysctl_net_ipv4.c=0A= +++ b/net/ipv4/sysctl_net_ipv4.c=0A= @@ -589,6 +589,20 @@ static struct ctl_table ipv4_table[] =3D {=0A= .mode =3D 0644,=0A= .proc_handler =3D proc_dointvec=0A= },=0A= + {=0A= + .procname =3D "tcp_limit_output_ms",=0A= + .data =3D &sysctl_tcp_limit_output_ms,=0A= + .maxlen =3D sizeof(unsigned int),=0A= + .mode =3D 0644,=0A= + .proc_handler =3D proc_douintvec=0A= + },=0A= + {=0A= + .procname =3D "tcp_limit_output_pkt",=0A= + .data =3D &sysctl_tcp_limit_output_pkt,=0A= + .maxlen =3D sizeof(unsigned int),=0A= + .mode =3D 0644,=0A= + .proc_handler =3D proc_douintvec=0A= + },=0A= {=0A= .procname =3D "tcp_challenge_ack_limit",=0A= .data =3D &sysctl_tcp_challenge_ack_limit,=0A= diff --git a/net/ipv4/tcp_output.c b/net/ipv4/tcp_output.c=0A= index 40f7c8ee9ba6..d06f1ae6950d 100644=0A= --- a/net/ipv4/tcp_output.c=0A= +++ b/net/ipv4/tcp_output.c=0A= @@ -53,6 +53,12 @@ int sysctl_tcp_workaround_signed_windows = __read_mostly =3D 0;=0A= /* Default TSQ limit of four TSO segments */=0A= int sysctl_tcp_limit_output_bytes __read_mostly =3D 262144;=0A= =0A= +/* Default TSQ limit of 1 ms of data, if the rate is set */=0A= +unsigned int sysctl_tcp_limit_output_ms __read_mostly =3D 1;=0A= +=0A= +/* Default TSQ limit of 2 packets */=0A= +unsigned int sysctl_tcp_limit_output_pkt __read_mostly =3D 2;=0A= +=0A= /* This limits the percentage of the congestion window which we=0A= * will allow a single TSO frame to consume. Building TSO frames=0A= * which are too large can cause TCP streams to be bursty.=0A= @@ -2176,7 +2182,11 @@ static bool tcp_small_queue_check(struct sock = *sk, const struct sk_buff *skb,=0A= {=0A= unsigned int limit;=0A= =0A= - limit =3D max(2 * skb->truesize, sk->sk_pacing_rate >> 10);=0A= + if (sysctl_tcp_limit_output_bytes < 0)=0A= + return false;=0A= +=0A= + limit =3D sysctl_tcp_limit_output_ms * (sk->sk_pacing_rate >> 10);=0A= + limit =3D max(sysctl_tcp_limit_output_pkt * skb->truesize, limit);=0A= limit =3D min_t(u32, limit, sysctl_tcp_limit_output_bytes);=0A= limit <<=3D factor;=0A= =0A= ------=_NextPart_000_003E_01D8FFEF.13A66C10--