[Bloat] Aggregating without bloating - hard times for tcp and wifi

[Bloat] Aggregating without bloating - hard times for tcp and wifi

[Dave Taht]

This paper came out last month. Good work, really exhaustive look at
two chipsets, multiple congestion controls and the interactions with
TSQ, with
lots and lots of flent.

https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=9772053

as for wifi6... don't make me start talking about wifi6... but some of
these tests look like a good baseline to start comparing the ath11k,
mt79, etc..

Paper kind of misses the negative impact of AQL in the ath10k (and
most likely also the mt76 and mt79 chips)

-- 
This song goes out to all the folk that thought Stadia would work:
https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz
Dave Täht CEO, TekLibre, LLC

Re: [Bloat] [bbr-dev] Aggregating without bloating - hard times for tcp and wifi

[Bob McMahon]

[-- Attachment #1.1: Type: text/plain, Size: 3144 bytes --]

Thanks for sharing this. Curious about how the xTSQ value can be set? Can
it be done with sysctl?

*We continue our analysis by using the ms-version of TSQ patch, which
enables the tune of the TSQ size allowing each TCP variant to enqueue more
than 1 ms of data at the current TCP rate. In particular, we allow to
enqueue the equivalent of x ms of data, naming each test xTSQ, with x being
an integer value. It is important to notice that this patch has been
included in the Linux kernel mainline, and each Wi-Fi driver can now set
the desired xTSQ value**.*

Another thing that could be interesting is the WiFi aggregation stop
reasons, e.g. how many times agg stopped per hitting the max mpdu per ampdu
vs the software fifo going empty (i.e. no more packets available to the
driver from the TCP stack) per that TXOP.

Finally, many (most?) APs are forwarding and feeding packets at at the
hardware level so not sure that the linux stack matters as much for an AP
based analysis, particularly when considering multi user transmissions,
i.e. multiple WiFi clients are active and sharing TXOPs.

Bob
On Mon, Nov 21, 2022 at 10:04 PM Dave Taht <dave.taht@gmail.com> wrote:

> This paper came out last month. Good work, really exhaustive look at
> two chipsets, multiple congestion controls and the interactions with
> TSQ, with
> lots and lots of flent.
>
> https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=9772053
>
> as for wifi6... don't make me start talking about wifi6... but some of
> these tests look like a good baseline to start comparing the ath11k,
> mt79, etc..
>
> Paper kind of misses the negative impact of AQL in the ath10k (and
> most likely also the mt76 and mt79 chips)
>
> --
> This song goes out to all the folk that thought Stadia would work:
>
> https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz
> Dave Täht CEO, TekLibre, LLC
>
> --
> You received this message because you are subscribed to the Google Groups
> "BBR Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to bbr-dev+unsubscribe@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/bbr-dev/CAA93jw6yJU10wh9ajqX4yW2AvnJPStyWAAcV4%2BoQ8wSGsJgKZA%40mail.gmail.com
> .
>

-- 
This electronic communication and the information and any files transmitted 
with it, or attached to it, are confidential and are intended solely for 
the use of the individual or entity to whom it is addressed and may contain 
information that is confidential, legally privileged, protected by privacy 
laws, or otherwise restricted from disclosure to anyone else. If you are 
not the intended recipient or the person responsible for delivering the 
e-mail to the intended recipient, you are hereby notified that any use, 
copying, distributing, dissemination, forwarding, printing, or copying of 
this e-mail is strictly prohibited. If you received this e-mail in error, 
please return the e-mail to the sender, delete it from your computer, and 
destroy any printed copy of it.

[-- Attachment #1.2: Type: text/html, Size: 4132 bytes --]

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4206 bytes --]

Re: [Bloat] [Make-wifi-fast] [bbr-dev] Aggregating without bloating - hard times for tcp and wifi

[David Lang]

On Tue, 22 Nov 2022, Bob McMahon via Make-wifi-fast wrote:

> Finally, many (most?) APs are forwarding and feeding packets at at the
> hardware level so not sure that the linux stack matters as much for an AP
> based analysis, particularly when considering multi user transmissions,
> i.e. multiple WiFi clients are active and sharing TXOPs.

APs forward packets within the switch at the hardware level, but the radios have 
to go through the CPU, so any wired <-> wireless needs to go through the CPU, 
and I would be incredibly surprised if the wifi chips did wireless <-> wireless 
routing at the hardware level.

David Lang

Re: [Bloat] [Make-wifi-fast] [bbr-dev] Aggregating without bloating - hard times for tcp and wifi

[Bob McMahon]

[-- Attachment #1.1: Type: text/plain, Size: 1853 bytes --]

An AP's radio complex may have a CPU but that doesn't mean it is the
standard linux stack as most think of it. Many consider this as part of
"firmware" which can be Linux, a Linux derivative or other.  Also, there
are some levels of wired/wireless forwarding plane integration done at the
hardware level that many might be surprised by.

Bob

On Tue, Nov 22, 2022 at 12:03 PM David Lang <david@lang.hm> wrote:

> On Tue, 22 Nov 2022, Bob McMahon via Make-wifi-fast wrote:
>
> > Finally, many (most?) APs are forwarding and feeding packets at at the
> > hardware level so not sure that the linux stack matters as much for an AP
> > based analysis, particularly when considering multi user transmissions,
> > i.e. multiple WiFi clients are active and sharing TXOPs.
>
> APs forward packets within the switch at the hardware level, but the
> radios have
> to go through the CPU, so any wired <-> wireless needs to go through the
> CPU,
> and I would be incredibly surprised if the wifi chips did wireless <->
> wireless
> routing at the hardware level.
>
> David Lang
>

-- 
This electronic communication and the information and any files transmitted 
with it, or attached to it, are confidential and are intended solely for 
the use of the individual or entity to whom it is addressed and may contain 
information that is confidential, legally privileged, protected by privacy 
laws, or otherwise restricted from disclosure to anyone else. If you are 
not the intended recipient or the person responsible for delivering the 
e-mail to the intended recipient, you are hereby notified that any use, 
copying, distributing, dissemination, forwarding, printing, or copying of 
this e-mail is strictly prohibited. If you received this e-mail in error, 
please return the e-mail to the sender, delete it from your computer, and 
destroy any printed copy of it.

[-- Attachment #1.2: Type: text/html, Size: 2286 bytes --]

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4206 bytes --]

Re: [Bloat] [Make-wifi-fast] [bbr-dev] Aggregating without bloating - hard times for tcp and wifi

[David Lang]

sorry, when I was saying 'the cpu', I was meaning the main one running linux, 
not something that's part of the wifi chipset.

I would be very surprised if the wifi chipset is doing any packet routing, as 
opposed to just sending the packets to the main processor.

Remember, the common case isn't forwarding from one wifi device to another, it's 
moving between wifi devices and the wired uplink.

David Lang

On Tue, 22 Nov 2022, Bob McMahon wrote:

> An AP's radio complex may have a CPU but that doesn't mean it is the
> standard linux stack as most think of it. Many consider this as part of
> "firmware" which can be Linux, a Linux derivative or other.  Also, there
> are some levels of wired/wireless forwarding plane integration done at the
> hardware level that many might be surprised by.
>
> Bob
>
> On Tue, Nov 22, 2022 at 12:03 PM David Lang <david@lang.hm> wrote:
>
>> On Tue, 22 Nov 2022, Bob McMahon via Make-wifi-fast wrote:
>>
>>> Finally, many (most?) APs are forwarding and feeding packets at at the
>>> hardware level so not sure that the linux stack matters as much for an AP
>>> based analysis, particularly when considering multi user transmissions,
>>> i.e. multiple WiFi clients are active and sharing TXOPs.
>>
>> APs forward packets within the switch at the hardware level, but the
>> radios have
>> to go through the CPU, so any wired <-> wireless needs to go through the
>> CPU,
>> and I would be incredibly surprised if the wifi chips did wireless <->
>> wireless
>> routing at the hardware level.
>>
>> David Lang
>>
>
>

Re: [Bloat] [Make-wifi-fast] [bbr-dev] Aggregating without bloating - hard times for tcp and wifi

[Bob McMahon]

[-- Attachment #1.1: Type: text/plain, Size: 2977 bytes --]

Some main purposes of the WiFi CPU is 802.3 to 802.11 L2 translational
bridging and handling 802.11 protocols for things like association. Most
forwarded packets don't hit the main CPU anymore. This first sw to hw
transition occurred decades ago with real internet routers (equipment that
run IGPs and BGP) which started as software in the early 90s and then moved
to hardware. The same engineering has been happening for home gateways or
WiFi APs bridging wired to wireless.

Bob

On Tue, Nov 22, 2022 at 12:16 PM David Lang <david@lang.hm> wrote:

> sorry, when I was saying 'the cpu', I was meaning the main one running
> linux,
> not something that's part of the wifi chipset.
>
> I would be very surprised if the wifi chipset is doing any packet routing,
> as
> opposed to just sending the packets to the main processor.
>
> Remember, the common case isn't forwarding from one wifi device to
> another, it's
> moving between wifi devices and the wired uplink.
>
> David Lang
>
> On Tue, 22 Nov 2022, Bob McMahon wrote:
>
> > An AP's radio complex may have a CPU but that doesn't mean it is the
> > standard linux stack as most think of it. Many consider this as part of
> > "firmware" which can be Linux, a Linux derivative or other.  Also, there
> > are some levels of wired/wireless forwarding plane integration done at
> the
> > hardware level that many might be surprised by.
> >
> > Bob
> >
> > On Tue, Nov 22, 2022 at 12:03 PM David Lang <david@lang.hm> wrote:
> >
> >> On Tue, 22 Nov 2022, Bob McMahon via Make-wifi-fast wrote:
> >>
> >>> Finally, many (most?) APs are forwarding and feeding packets at at the
> >>> hardware level so not sure that the linux stack matters as much for an
> AP
> >>> based analysis, particularly when considering multi user transmissions,
> >>> i.e. multiple WiFi clients are active and sharing TXOPs.
> >>
> >> APs forward packets within the switch at the hardware level, but the
> >> radios have
> >> to go through the CPU, so any wired <-> wireless needs to go through the
> >> CPU,
> >> and I would be incredibly surprised if the wifi chips did wireless <->
> >> wireless
> >> routing at the hardware level.
> >>
> >> David Lang
> >>
> >
> >
>

-- 
This electronic communication and the information and any files transmitted 
with it, or attached to it, are confidential and are intended solely for 
the use of the individual or entity to whom it is addressed and may contain 
information that is confidential, legally privileged, protected by privacy 
laws, or otherwise restricted from disclosure to anyone else. If you are 
not the intended recipient or the person responsible for delivering the 
e-mail to the intended recipient, you are hereby notified that any use, 
copying, distributing, dissemination, forwarding, printing, or copying of 
this e-mail is strictly prohibited. If you received this e-mail in error, 
please return the e-mail to the sender, delete it from your computer, and 
destroy any printed copy of it.

[-- Attachment #1.2: Type: text/html, Size: 3714 bytes --]

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4206 bytes --]

Re: [Bloat] [Make-wifi-fast] [bbr-dev] Aggregating without bloating - hard times for tcp and wifi

[Bob McMahon]

[-- Attachment #1.1: Type: text/plain, Size: 3639 bytes --]

I don't know Qualcomm's offerings but here are some from Broadcom.

https://www.broadcom.com/products/wireless/wireless-lan-infrastructure/bcm67263

The BCM4916 forwarding plane is done with a network processor and doesn't
run Linux.  Linux may be used to build the forwarding tables and this is
standard "merchant silicon" forwarding approcch, let some CPU/stack build
the topology tables and then realize the packet forwarding in
(programmable) hardware.

https://docs.broadcom.com/doc/4916-PB1XX

Bob

On Tue, Nov 22, 2022 at 12:28 PM Bob McMahon <bob.mcmahon@broadcom.com>
wrote:

> Some main purposes of the WiFi CPU is 802.3 to 802.11 L2 translational
> bridging and handling 802.11 protocols for things like association. Most
> forwarded packets don't hit the main CPU anymore. This first sw to hw
> transition occurred decades ago with real internet routers (equipment that
> run IGPs and BGP) which started as software in the early 90s and then moved
> to hardware. The same engineering has been happening for home gateways or
> WiFi APs bridging wired to wireless.
>
> Bob
>
> On Tue, Nov 22, 2022 at 12:16 PM David Lang <david@lang.hm> wrote:
>
>> sorry, when I was saying 'the cpu', I was meaning the main one running
>> linux,
>> not something that's part of the wifi chipset.
>>
>> I would be very surprised if the wifi chipset is doing any packet
>> routing, as
>> opposed to just sending the packets to the main processor.
>>
>> Remember, the common case isn't forwarding from one wifi device to
>> another, it's
>> moving between wifi devices and the wired uplink.
>>
>> David Lang
>>
>> On Tue, 22 Nov 2022, Bob McMahon wrote:
>>
>> > An AP's radio complex may have a CPU but that doesn't mean it is the
>> > standard linux stack as most think of it. Many consider this as part of
>> > "firmware" which can be Linux, a Linux derivative or other.  Also, there
>> > are some levels of wired/wireless forwarding plane integration done at
>> the
>> > hardware level that many might be surprised by.
>> >
>> > Bob
>> >
>> > On Tue, Nov 22, 2022 at 12:03 PM David Lang <david@lang.hm> wrote:
>> >
>> >> On Tue, 22 Nov 2022, Bob McMahon via Make-wifi-fast wrote:
>> >>
>> >>> Finally, many (most?) APs are forwarding and feeding packets at at the
>> >>> hardware level so not sure that the linux stack matters as much for
>> an AP
>> >>> based analysis, particularly when considering multi user
>> transmissions,
>> >>> i.e. multiple WiFi clients are active and sharing TXOPs.
>> >>
>> >> APs forward packets within the switch at the hardware level, but the
>> >> radios have
>> >> to go through the CPU, so any wired <-> wireless needs to go through
>> the
>> >> CPU,
>> >> and I would be incredibly surprised if the wifi chips did wireless <->
>> >> wireless
>> >> routing at the hardware level.
>> >>
>> >> David Lang
>> >>
>> >
>> >
>>
>

-- 
This electronic communication and the information and any files transmitted 
with it, or attached to it, are confidential and are intended solely for 
the use of the individual or entity to whom it is addressed and may contain 
information that is confidential, legally privileged, protected by privacy 
laws, or otherwise restricted from disclosure to anyone else. If you are 
not the intended recipient or the person responsible for delivering the 
e-mail to the intended recipient, you are hereby notified that any use, 
copying, distributing, dissemination, forwarding, printing, or copying of 
this e-mail is strictly prohibited. If you received this e-mail in error, 
please return the e-mail to the sender, delete it from your computer, and 
destroy any printed copy of it.

[-- Attachment #1.2: Type: text/html, Size: 4800 bytes --]

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4206 bytes --]

Re: [Bloat] [bbr-dev] Aggregating without bloating - hard times for tcp and wifi

[Neal Cardwell]

[-- Attachment #1: Type: text/plain, Size: 4293 bytes --]

On Tue, Nov 22, 2022 at 2:43 PM 'Bob McMahon' via BBR Development <
bbr-dev@googlegroups.com> wrote:

> Thanks for sharing this. Curious about how the xTSQ value can be set? Can
> it be done with sysctl?
>
> *We continue our analysis by using the ms-version of TSQ patch, which
> enables the tune of the TSQ size allowing each TCP variant to enqueue more
> than 1 ms of data at the current TCP rate. In particular, we allow to
> enqueue the equivalent of x ms of data, naming each test xTSQ, with x being
> an integer value. It is important to notice that this patch has been
> included in the Linux kernel mainline, and each Wi-Fi driver can now set
> the desired xTSQ value**.*
>

I believe they are setting the xTSQ value using the sk_pacing_shift field,
which was added here:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=3a9b76fd0db9f0d426533f96a68a62a58753a51e

AFAIK the intent is only for drivers to set that, and there's no sysctl for
that, but of course you could add a sysctl for testing if you wanted. :-)

cheers,
neal




> Another thing that could be interesting is the WiFi aggregation stop
> reasons, e.g. how many times agg stopped per hitting the max mpdu per ampdu
> vs the software fifo going empty (i.e. no more packets available to the
> driver from the TCP stack) per that TXOP.
>
> Finally, many (most?) APs are forwarding and feeding packets at at the
> hardware level so not sure that the linux stack matters as much for an AP
> based analysis, particularly when considering multi user transmissions,
> i.e. multiple WiFi clients are active and sharing TXOPs.
>
> Bob
> On Mon, Nov 21, 2022 at 10:04 PM Dave Taht <dave.taht@gmail.com> wrote:
>
>> This paper came out last month. Good work, really exhaustive look at
>> two chipsets, multiple congestion controls and the interactions with
>> TSQ, with
>> lots and lots of flent.
>>
>> https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=9772053
>>
>> as for wifi6... don't make me start talking about wifi6... but some of
>> these tests look like a good baseline to start comparing the ath11k,
>> mt79, etc..
>>
>> Paper kind of misses the negative impact of AQL in the ath10k (and
>> most likely also the mt76 and mt79 chips)
>>
>> --
>> This song goes out to all the folk that thought Stadia would work:
>>
>> https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz
>> Dave Täht CEO, TekLibre, LLC
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "BBR Development" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to bbr-dev+unsubscribe@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/bbr-dev/CAA93jw6yJU10wh9ajqX4yW2AvnJPStyWAAcV4%2BoQ8wSGsJgKZA%40mail.gmail.com
>> .
>>
>
> This electronic communication and the information and any files
> transmitted with it, or attached to it, are confidential and are intended
> solely for the use of the individual or entity to whom it is addressed and
> may contain information that is confidential, legally privileged, protected
> by privacy laws, or otherwise restricted from disclosure to anyone else. If
> you are not the intended recipient or the person responsible for delivering
> the e-mail to the intended recipient, you are hereby notified that any use,
> copying, distributing, dissemination, forwarding, printing, or copying of
> this e-mail is strictly prohibited. If you received this e-mail in error,
> please return the e-mail to the sender, delete it from your computer, and
> destroy any printed copy of it.
>
> --
> You received this message because you are subscribed to the Google Groups
> "BBR Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to bbr-dev+unsubscribe@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/bbr-dev/CAHb6LvrK_exkt_rbukYkF4_hwPXyy9T%2BZNGzdK79x91RrK49Mg%40mail.gmail.com
> <https://groups.google.com/d/msgid/bbr-dev/CAHb6LvrK_exkt_rbukYkF4_hwPXyy9T%2BZNGzdK79x91RrK49Mg%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>

[-- Attachment #2: Type: text/html, Size: 6010 bytes --]

Re: [Bloat] [bbr-dev] Aggregating without bloating - hard times for tcp and wifi

[Toke Høiland-Jørgensen]

Neal Cardwell via Bloat <bloat@lists.bufferbloat.net> writes:

> On Tue, Nov 22, 2022 at 2:43 PM 'Bob McMahon' via BBR Development <
> bbr-dev@googlegroups.com> wrote:
>
>> Thanks for sharing this. Curious about how the xTSQ value can be set? Can
>> it be done with sysctl?
>>
>> *We continue our analysis by using the ms-version of TSQ patch, which
>> enables the tune of the TSQ size allowing each TCP variant to enqueue more
>> than 1 ms of data at the current TCP rate. In particular, we allow to
>> enqueue the equivalent of x ms of data, naming each test xTSQ, with x being
>> an integer value. It is important to notice that this patch has been
>> included in the Linux kernel mainline, and each Wi-Fi driver can now set
>> the desired xTSQ value**.*
>>
>
> I believe they are setting the xTSQ value using the sk_pacing_shift field,
> which was added here:
>
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=3a9b76fd0db9f0d426533f96a68a62a58753a51e
>
> AFAIK the intent is only for drivers to set that, and there's no sysctl for
> that, but of course you could add a sysctl for testing if you wanted.
> :-)

Yup, indeed this is what mac80211 fiddles with:
https://elixir.bootlin.com/linux/latest/source/net/mac80211/main.c#L739
https://elixir.bootlin.com/linux/latest/source/net/mac80211/tx.c#L4156

AFAICT, no in-tree drivers override the value set by mac80211.

I believe the tests in that paper were conducted with this series
applied:
https://lore.kernel.org/all/20180105113256.14835-1-natale.patriciello@gmail.com/

-Toke

Re: [Bloat] [bbr-dev] Aggregating without bloating - hard times for tcp and wifi

[Bob McMahon]

[-- Attachment #1.1: Type: text/plain, Size: 2816 bytes --]

Does the TSQ code honor no-aggregation per voice access class or
TCP_NODELAY where the app making the socket write calls knows that the WiFi
aggregation isn't likely helpful? Sorry, my Linux stack expertise is quite
limited.

Bob

On Tue, Nov 22, 2022 at 12:53 PM Toke Høiland-Jørgensen <toke@toke.dk>
wrote:

> Neal Cardwell via Bloat <bloat@lists.bufferbloat.net> writes:
>
> > On Tue, Nov 22, 2022 at 2:43 PM 'Bob McMahon' via BBR Development <
> > bbr-dev@googlegroups.com> wrote:
> >
> >> Thanks for sharing this. Curious about how the xTSQ value can be set?
> Can
> >> it be done with sysctl?
> >>
> >> *We continue our analysis by using the ms-version of TSQ patch, which
> >> enables the tune of the TSQ size allowing each TCP variant to enqueue
> more
> >> than 1 ms of data at the current TCP rate. In particular, we allow to
> >> enqueue the equivalent of x ms of data, naming each test xTSQ, with x
> being
> >> an integer value. It is important to notice that this patch has been
> >> included in the Linux kernel mainline, and each Wi-Fi driver can now set
> >> the desired xTSQ value**.*
> >>
> >
> > I believe they are setting the xTSQ value using the sk_pacing_shift
> field,
> > which was added here:
> >
> >
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=3a9b76fd0db9f0d426533f96a68a62a58753a51e
> >
> > AFAIK the intent is only for drivers to set that, and there's no sysctl
> for
> > that, but of course you could add a sysctl for testing if you wanted.
> > :-)
>
> Yup, indeed this is what mac80211 fiddles with:
> https://elixir.bootlin.com/linux/latest/source/net/mac80211/main.c#L739
> https://elixir.bootlin.com/linux/latest/source/net/mac80211/tx.c#L4156
>
> AFAICT, no in-tree drivers override the value set by mac80211.
>
> I believe the tests in that paper were conducted with this series
> applied:
>
> https://lore.kernel.org/all/20180105113256.14835-1-natale.patriciello@gmail.com/
>
> -Toke
>

-- 
This electronic communication and the information and any files transmitted 
with it, or attached to it, are confidential and are intended solely for 
the use of the individual or entity to whom it is addressed and may contain 
information that is confidential, legally privileged, protected by privacy 
laws, or otherwise restricted from disclosure to anyone else. If you are 
not the intended recipient or the person responsible for delivering the 
e-mail to the intended recipient, you are hereby notified that any use, 
copying, distributing, dissemination, forwarding, printing, or copying of 
this e-mail is strictly prohibited. If you received this e-mail in error, 
please return the e-mail to the sender, delete it from your computer, and 
destroy any printed copy of it.

[-- Attachment #1.2: Type: text/html, Size: 3984 bytes --]

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4206 bytes --]

Re: [Bloat] [bbr-dev] Aggregating without bloating - hard times for tcp and wifi

[Toke Høiland-Jørgensen]

Bob McMahon <bob.mcmahon@broadcom.com> writes:

> Does the TSQ code honor no-aggregation per voice access class or
> TCP_NODELAY where the app making the socket write calls knows that the WiFi
> aggregation isn't likely helpful? Sorry, my Linux stack expertise is quite
> limited.

TSQ only influences the buffering in the TCP layer. The WiFi stack will
still limit aggregation using its own logic (I think it turns it off
entirely for voice?). TCP_NODELAY is also orthogonal to TSQ; TSQ only
kicks in when there's a bunch of data buffered, in which case
TCP_NODELAY has no effect...

-Toke

Re: [Bloat] [bbr-dev] Aggregating without bloating - hard times for tcp and wifi

[Bob McMahon]

[-- Attachment #1.1: Type: text/plain, Size: 1543 bytes --]

Thanks Toke.

Bob

On Wed, Nov 23, 2022 at 5:50 AM Toke Høiland-Jørgensen <toke@toke.dk> wrote:

> Bob McMahon <bob.mcmahon@broadcom.com> writes:
>
> > Does the TSQ code honor no-aggregation per voice access class or
> > TCP_NODELAY where the app making the socket write calls knows that the
> WiFi
> > aggregation isn't likely helpful? Sorry, my Linux stack expertise is
> quite
> > limited.
>
> TSQ only influences the buffering in the TCP layer. The WiFi stack will
> still limit aggregation using its own logic (I think it turns it off
> entirely for voice?). TCP_NODELAY is also orthogonal to TSQ; TSQ only
> kicks in when there's a bunch of data buffered, in which case
> TCP_NODELAY has no effect...
>
> -Toke
>

-- 
This electronic communication and the information and any files transmitted 
with it, or attached to it, are confidential and are intended solely for 
the use of the individual or entity to whom it is addressed and may contain 
information that is confidential, legally privileged, protected by privacy 
laws, or otherwise restricted from disclosure to anyone else. If you are 
not the intended recipient or the person responsible for delivering the 
e-mail to the intended recipient, you are hereby notified that any use, 
copying, distributing, dissemination, forwarding, printing, or copying of 
this e-mail is strictly prohibited. If you received this e-mail in error, 
please return the e-mail to the sender, delete it from your computer, and 
destroy any printed copy of it.

[-- Attachment #1.2: Type: text/html, Size: 2091 bytes --]

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4206 bytes --]

Re: [Bloat] [bbr-dev] Aggregating without bloating - hard times for tcp and wifi

[Dave Taht]

On Wed, Nov 23, 2022 at 9:25 PM Muhammad Ahsan <muhammadahsan@umt.edu.pk> wrote:
>
> Hi dev group guys,
>
>
>
> I need to use sysctl to set ms value for net.ipv4.tcp_limit_output_ms .
>
>
>
> The TSQ patch attached is not working or letting me do that . I am on linux 5.13.12

Why would you test using a 2 year old kernel? In general I try to
compile a net-next or a rc3+ candidate when doing network research. By
the time your paper is published, your
work is 3 or more years obsolete. Working within a modern kernel you
can get in general get help from the netdev mailing list.

Admittedly, embedded OSes like openwrt tend to lag a few years behind
also (presently at 5.15 for most chipsets).


> Manually changing tx_sk_pacing_shift = 7;  variable in main.c  needs to recompile kernel everytime…
>
>
>
> I need to have sysctl to control the ms value , to set  2TSQ,4TSQ,8TSQ  etc for my wifi chipset.

One of the reasons why I was reluctant to provide this knob was that
it seemed inevitable folk would optimize for bandwidth at all other
costs, in a lab environment. We picked numbers for the initial
implementations of this that made a good compromise between latency
and throughput. Certainly tuning it up would be good, but *first* I
try to encourage researchers to emulate real-world conditions, where
interference and other stations can cause txop scheduling delays
measured in the 100s or 1000s of ms. The flent rtt_fair test, to 4 or
more stations, in particular, can be quite revealing and our test
results and test benchmarks are here:

https://www.cs.kau.se/tohojo/airtime-fairness/

There is MUCH other low hanging fruit in wifi. As one example, if you
look at aircaps, you will find a lot of single packet, followed by an
aggregate, because the driver accepts and schedules that first packet
(and txop) and doesn't wait a little bit  to assemble a larger
aggregate, essentially wasting a txop. There is still, 7+ years since
this preso of mine:
https://www.youtube.com/watch?v=Rb-UnHDw02o&t=1591s not a lot of
understanding of how aggregation behaves. Furthermore ack-filtering is
proving helpful in many
scenarios.

I am sad to report that the initial mt79 firmware design does not
expose per station queues, which are IMHO utterly necessary for low
latency in wifi6 and wifi7. I have high hopes for the new qualcomm
chips...

Anyway, some more links:

https://forum.openwrt.org/t/reducing-multiplexing-latencies-still-further-in-wifi/133605/

https://forum.openwrt.org/t/aql-and-the-ath10k-is-lovely/59002

While I think that most of the patches I care about are in your 5.13
version, there's stuff scheduled for 6.2 or 6.3, and the AQL mechanism
used by the ath10k, mt76, iwl, and now mt79, is proving to be a real
barrier to good throughput and latency over wifi.




>
>
>
> I will be thankful if anyone can help me in it.
>
>
>
>
>
> Rgds,
>
> Ahsan
>
>
>
>
>
>
>
> From: 'Neal Cardwell' via BBR Development <bbr-dev@googlegroups.com>
> Sent: Wednesday, November 23, 2022 1:10 AM
> To: Bob McMahon <bob.mcmahon@broadcom.com>
> Cc: Dave Taht <dave.taht@gmail.com>; Make-Wifi-fast <make-wifi-fast@lists.bufferbloat.net>; bloat <bloat@lists.bufferbloat.net>; BBR Development <bbr-dev@googlegroups.com>
> Subject: Re: [bbr-dev] Aggregating without bloating - hard times for tcp and wifi
>
>
>
> On Tue, Nov 22, 2022 at 2:43 PM 'Bob McMahon' via BBR Development <bbr-dev@googlegroups.com> wrote:
>
> Thanks for sharing this. Curious about how the xTSQ value can be set? Can it be done with sysctl?
>
> We continue our analysis by using the ms-version of TSQ patch, which enables the tune of the TSQ size allowing each TCP variant to enqueue more than 1 ms of data at the current TCP rate. In particular, we allow to enqueue the equivalent of x ms of data, naming each test xTSQ, with x being an integer value. It is important to notice that this patch has been included in the Linux kernel mainline, and each Wi-Fi driver can now set the desired xTSQ value.
>
>
>
> I believe they are setting the xTSQ value using the sk_pacing_shift field, which was added here:
>
>    https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=3a9b76fd0db9f0d426533f96a68a62a58753a51e
>
>
>
> AFAIK the intent is only for drivers to set that, and there's no sysctl for that, but of course you could add a sysctl for testing if you wanted. :-)
>
>
>
> cheers,
>
> neal
>
>
>
>
>
>
>
> Another thing that could be interesting is the WiFi aggregation stop reasons, e.g. how many times agg stopped per hitting the max mpdu per ampdu vs the software fifo going empty (i.e. no more packets available to the driver from the TCP stack) per that TXOP.
>
> Finally, many (most?) APs are forwarding and feeding packets at at the hardware level so not sure that the linux stack matters as much for an AP based analysis, particularly when considering multi user transmissions, i.e. multiple WiFi clients are active and sharing TXOPs.
>
> Bob
>
> On Mon, Nov 21, 2022 at 10:04 PM Dave Taht <dave.taht@gmail.com> wrote:
>
> This paper came out last month. Good work, really exhaustive look at
> two chipsets, multiple congestion controls and the interactions with
> TSQ, with
> lots and lots of flent.
>
> https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=9772053
>
> as for wifi6... don't make me start talking about wifi6... but some of
> these tests look like a good baseline to start comparing the ath11k,
> mt79, etc..
>
> Paper kind of misses the negative impact of AQL in the ath10k (and
> most likely also the mt76 and mt79 chips)
>
> --
> This song goes out to all the folk that thought Stadia would work:
> https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz
> Dave Täht CEO, TekLibre, LLC
>
> --
> You received this message because you are subscribed to the Google Groups "BBR Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to bbr-dev+unsubscribe@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/bbr-dev/CAA93jw6yJU10wh9ajqX4yW2AvnJPStyWAAcV4%2BoQ8wSGsJgKZA%40mail.gmail.com.
>
>
> This electronic communication and the information and any files transmitted with it, or attached to it, are confidential and are intended solely for the use of the individual or entity to whom it is addressed and may contain information that is confidential, legally privileged, protected by privacy laws, or otherwise restricted from disclosure to anyone else. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended recipient, you are hereby notified that any use, copying, distributing, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited. If you received this e-mail in error, please return the e-mail to the sender, delete it from your computer, and destroy any printed copy of it.
>
> --
> You received this message because you are subscribed to the Google Groups "BBR Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to bbr-dev+unsubscribe@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/bbr-dev/CAHb6LvrK_exkt_rbukYkF4_hwPXyy9T%2BZNGzdK79x91RrK49Mg%40mail.gmail.com.
>
> --
> You received this message because you are subscribed to the Google Groups "BBR Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to bbr-dev+unsubscribe@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/bbr-dev/CADVnQyn9OSxPcF2JsUHdiWLTvAAdN5vs1uyiRBL0AS8yH8niYw%40mail.gmail.com.



-- 
This song goes out to all the folk that thought Stadia would work:
https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-6981366665607352320-FXtz
Dave Täht CEO, TekLibre, LLC