[Make-wifi-fast] lwn.net's tcp small queues vs wifi aggregation solved

[Make-wifi-fast] lwn.net's tcp small queues vs wifi aggregation solved

[Dave Taht]

Nice war story. I'm glad this last problem with the fq_codel wifi code
is solved, and the article points to a few usb wifi dongles that work
better now.

https://lwn.net/SubscriberLink/757643/b25587e3593e9f9e/



-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

Re: [Make-wifi-fast] [Bloat] lwn.net's tcp small queues vs wifi aggregation solved

[Toke Høiland-Jørgensen]

Dave Taht <dave.taht@gmail.com> writes:

> Nice war story. I'm glad this last problem with the fq_codel wifi code
> is solved

This wasn't specific to the fq_codel wifi code, but hit all WiFi devices
that were running TCP on the local stack. Which would be mostly laptops,
I guess...

-Toke

Re: [Make-wifi-fast] [Bloat] lwn.net's tcp small queues vs wifi aggregation solved

[Dave Taht]

On Thu, Jun 21, 2018 at 5:55 AM, Eric Dumazet <eric.dumazet@gmail.com> wrote:
>
>
> On 06/21/2018 02:22 AM, Toke Høiland-Jørgensen wrote:
>> Dave Taht <dave.taht@gmail.com> writes:
>>
>>> Nice war story. I'm glad this last problem with the fq_codel wifi code
>>> is solved
>>
>> This wasn't specific to the fq_codel wifi code, but hit all WiFi devices
>> that were running TCP on the local stack. Which would be mostly laptops,
>> I guess...
>
> Yes.
>
> Also switching TCP stack to always GSO has been a major gain for wifi in my tests.
>
> (TSQ budget is based on sk_wmem_alloc, tracking truesize of skbs, and not having
> GSO is considerably inflating the truesize/payload ratio)
>
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=0a6b2a1dc2a2105f178255fe495eb914b09cb37a
> tcp: switch to GSO being always on
>
> I expect SACK compression to also give a nice boost to wifi.
>
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=5d9f4262b7ea41ca9981cc790e37cca6e37c789e
> tcp: add SACK compression
>
> Lastly I am working on adding ACK compression in TCP stack itself.

One thing I've seen repeatedly on mac80211 aircaps is a tendency for
clients to use up two TXOPs rather than one.

scenario:

1) A tcp burst arrives at the client
2) A single ack migrates down the client stack into the driver, into
the device, which then attempts to compete for airtime on that TXOP
for that single ack, sometimes waiting 10s of msec to get that op
3) a bunch more acks arrive "slightly late"[1], and then get queued
for the next TXOP, waiting, again sometimes 10s of msec

(similar scenario in a client making a quick string of web related requests)

This is a case where inserting a teeny bit more latency to fill up the
queue (ugh!), or a driver having some way to ask the probability of
seeing more data in the
next 10us, or... something like that, could help.

...

[1] if you need coffee through your nose this morning, regarding usage
of the phrase "slightly late", read
http://www.rawbw.com/~svw/superman.html

-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

Re: [Make-wifi-fast] [Bloat] lwn.net's tcp small queues vs wifi aggregation solved

[Caleb Cushing]

[-- Attachment #1: Type: text/plain, Size: 2785 bytes --]

actually... all of my devices, including my desktop connect through wifi
these days... and only one of them isn't running some variant of linux.

On Thu, Jun 21, 2018 at 10:18 AM Dave Taht <dave.taht@gmail.com> wrote:

> On Thu, Jun 21, 2018 at 5:55 AM, Eric Dumazet <eric.dumazet@gmail.com>
> wrote:
> >
> >
> > On 06/21/2018 02:22 AM, Toke Høiland-Jørgensen wrote:
> >> Dave Taht <dave.taht@gmail.com> writes:
> >>
> >>> Nice war story. I'm glad this last problem with the fq_codel wifi code
> >>> is solved
> >>
> >> This wasn't specific to the fq_codel wifi code, but hit all WiFi devices
> >> that were running TCP on the local stack. Which would be mostly laptops,
> >> I guess...
> >
> > Yes.
> >
> > Also switching TCP stack to always GSO has been a major gain for wifi in
> my tests.
> >
> > (TSQ budget is based on sk_wmem_alloc, tracking truesize of skbs, and
> not having
> > GSO is considerably inflating the truesize/payload ratio)
> >
> >
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=0a6b2a1dc2a2105f178255fe495eb914b09cb37a
> > tcp: switch to GSO being always on
> >
> > I expect SACK compression to also give a nice boost to wifi.
> >
> >
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=5d9f4262b7ea41ca9981cc790e37cca6e37c789e
> > tcp: add SACK compression
> >
> > Lastly I am working on adding ACK compression in TCP stack itself.
>
> One thing I've seen repeatedly on mac80211 aircaps is a tendency for
> clients to use up two TXOPs rather than one.
>
> scenario:
>
> 1) A tcp burst arrives at the client
> 2) A single ack migrates down the client stack into the driver, into
> the device, which then attempts to compete for airtime on that TXOP
> for that single ack, sometimes waiting 10s of msec to get that op
> 3) a bunch more acks arrive "slightly late"[1], and then get queued
> for the next TXOP, waiting, again sometimes 10s of msec
>
> (similar scenario in a client making a quick string of web related
> requests)
>
> This is a case where inserting a teeny bit more latency to fill up the
> queue (ugh!), or a driver having some way to ask the probability of
> seeing more data in the
> next 10us, or... something like that, could help.
>
> ...
>
> [1] if you need coffee through your nose this morning, regarding usage
> of the phrase "slightly late", read
> http://www.rawbw.com/~svw/superman.html
>
> --
>
> Dave Täht
> CEO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-669-226-2619 <(669)%20226-2619>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat
>
-- 
Caleb Cushing

http://xenoterracide.com

[-- Attachment #2: Type: text/html, Size: 4279 bytes --]

Re: [Make-wifi-fast] [Bloat] lwn.net's tcp small queues vs wifi aggregation solved

[Stephen Hemminger]

On Thu, 21 Jun 2018 10:31:18 -0500
Caleb Cushing <xenoterracide@gmail.com> wrote:

> actually... all of my devices, including my desktop connect through wifi
> these days... and only one of them isn't running some variant of linux.
> 

Sigh. My experience with wifi is that it is not stable enough for that.
Both AP's I have tried Linksys ACM3200 or Netgear WNDR3800 I still see random drop outs.
Not sure if these are device resets (ie workarounds) or other issues.

These happen independent of firmware (vendor, OpenWRT, or LEDE).
So my suspicion is the that Wifi hardware is shite and that firmware is trying
to workaround and mask the problem.

Re: [Make-wifi-fast] [Bloat] lwn.net's tcp small queues vs wifi aggregation solved

[Caleb Cushing]

[-- Attachment #1: Type: text/plain, Size: 950 bytes --]

I'm not disagreeing, just saying that wifi is much more prevalent now than
just laptops... literally I only have a cord for emergency use

On Thu, Jun 21, 2018 at 10:46 AM Stephen Hemminger <
stephen@networkplumber.org> wrote:

> On Thu, 21 Jun 2018 10:31:18 -0500
> Caleb Cushing <xenoterracide@gmail.com> wrote:
>
> > actually... all of my devices, including my desktop connect through wifi
> > these days... and only one of them isn't running some variant of linux.
> >
>
> Sigh. My experience with wifi is that it is not stable enough for that.
> Both AP's I have tried Linksys ACM3200 or Netgear WNDR3800 I still see
> random drop outs.
> Not sure if these are device resets (ie workarounds) or other issues.
>
> These happen independent of firmware (vendor, OpenWRT, or LEDE).
> So my suspicion is the that Wifi hardware is shite and that firmware is
> trying
> to workaround and mask the problem.
>
-- 
Caleb Cushing

http://xenoterracide.com

[-- Attachment #2: Type: text/html, Size: 1493 bytes --]

Re: [Make-wifi-fast] [Bloat] lwn.net's tcp small queues vs wifi aggregation solved

[Dave Taht]

On Thu, Jun 21, 2018 at 8:31 AM, Caleb Cushing <xenoterracide@gmail.com> wrote:
> actually... all of my devices, including my desktop connect through wifi
> these days... and only one of them isn't running some variant of linux.

Yes the tendency of manufacturers to hook things up to the more
convenient, but overbuffered and less opaque USB bus has become an
increasingly large problem
(canonical example - raspberry pi). In the case of LTE, especially,
everything is a USB dongle, and the CDC_ETH driver and device spec
actually mandates at least 32k of
on-chip buffering on the other side of the bus.

We had tried at one point (5 years ago) to find ways to apply
something BQL-like to this but failed.

I am currently getting miserable performance out of the one LTE dongle
I have (16K/sec up) but have not gone and fiddled with it with more
modern kernels. I ended up
just tethering via an android phone, which cracks 1mbit up.

The quality of the wifi drivers for USB is almost uniformly miserable,
and out of tree.

>
> On Thu, Jun 21, 2018 at 10:18 AM Dave Taht <dave.taht@gmail.com> wrote:
>>
>> On Thu, Jun 21, 2018 at 5:55 AM, Eric Dumazet <eric.dumazet@gmail.com>
>> wrote:
>> >
>> >
>> > On 06/21/2018 02:22 AM, Toke Høiland-Jørgensen wrote:
>> >> Dave Taht <dave.taht@gmail.com> writes:
>> >>
>> >>> Nice war story. I'm glad this last problem with the fq_codel wifi code
>> >>> is solved
>> >>
>> >> This wasn't specific to the fq_codel wifi code, but hit all WiFi
>> >> devices
>> >> that were running TCP on the local stack. Which would be mostly
>> >> laptops,
>> >> I guess...
>> >
>> > Yes.
>> >
>> > Also switching TCP stack to always GSO has been a major gain for wifi in
>> > my tests.
>> >
>> > (TSQ budget is based on sk_wmem_alloc, tracking truesize of skbs, and
>> > not having
>> > GSO is considerably inflating the truesize/payload ratio)
>> >
>> >
>> > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=0a6b2a1dc2a2105f178255fe495eb914b09cb37a
>> > tcp: switch to GSO being always on
>> >
>> > I expect SACK compression to also give a nice boost to wifi.
>> >
>> >
>> > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=5d9f4262b7ea41ca9981cc790e37cca6e37c789e
>> > tcp: add SACK compression
>> >
>> > Lastly I am working on adding ACK compression in TCP stack itself.
>>
>> One thing I've seen repeatedly on mac80211 aircaps is a tendency for
>> clients to use up two TXOPs rather than one.
>>
>> scenario:
>>
>> 1) A tcp burst arrives at the client
>> 2) A single ack migrates down the client stack into the driver, into
>> the device, which then attempts to compete for airtime on that TXOP
>> for that single ack, sometimes waiting 10s of msec to get that op
>> 3) a bunch more acks arrive "slightly late"[1], and then get queued
>> for the next TXOP, waiting, again sometimes 10s of msec
>>
>> (similar scenario in a client making a quick string of web related
>> requests)
>>
>> This is a case where inserting a teeny bit more latency to fill up the
>> queue (ugh!), or a driver having some way to ask the probability of
>> seeing more data in the
>> next 10us, or... something like that, could help.
>>
>> ...
>>
>> [1] if you need coffee through your nose this morning, regarding usage
>> of the phrase "slightly late", read
>> http://www.rawbw.com/~svw/superman.html
>>
>> --
>>
>> Dave Täht
>> CEO, TekLibre, LLC
>> http://www.teklibre.com
>> Tel: 1-669-226-2619
>> _______________________________________________
>> Bloat mailing list
>> Bloat@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/bloat
>
> --
> Caleb Cushing
>
> http://xenoterracide.com



-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

Re: [Make-wifi-fast] [Bloat] lwn.net's tcp small queues vs wifi aggregation solved

[Dave Taht]

On Mon, Jun 25, 2018 at 5:44 PM, Jonathan Morton <chromatix99@gmail.com> wrote:
>> On 26 Jun, 2018, at 3:36 am, Simon Barber <simon@superduper.net> wrote:
>>
>> Most hardware needs the packet finalized before it starts to contend for the medium (as far as I’m aware - let me know if you know differently). One issue is that if RTS/CTS is in use, then the packet duration needs to be known in advance (or at least mid point of the RTS transmission).
>
> This is a valid argument.  I think we could successfully argue for a delay of 1ms, if there isn't already enough data in the queue to fill an aggregate, after the oldest packet arrives until a request is issued.

Whoa, nelly! In the context of the local tcp stack over wifi, I was
making an observation that I "frequently" saw a pattern of a single
ack txop followed by a bunch in a separate txop. and I suggested a
very short (10us) timeout before committing to the hw - not 1ms.

Aside from this anecdote we have not got real data or statistics. The
closest thing I have to a tool that can take apart wireless aircaps is
here: https://github.com/dtaht/airtime-pie-chart which can be hacked
to take more things apart than it currently does. Looking for this
pattern in more traffic would be revealing in multiple ways. Looking
for more patterns in bigger wifi networks would be good also.

I like erics suggestion of doing more ack compression higher up in the
tcp stack.

There are two other things I've suggested in the past we look at. 1)
The current fq_codel_for_wifi code has a philosophy of "one aggregate
in the hardware, one ready to go". A simpler modification to fit more
in would be to (wait the best case estimate for delivering the one in
the hardware - a bit), then form the one ready-to-go.

2) rate limiting mcast and smoothing mcast bursts over time, allowing
more unicast through. presently the mcast queue is infinite and very
bursty. 802.11 std actually suggests mcast be rate limited by htb,
where I'd be htb + fq + merging dup packets. I was routinely able to
blow up the c.h.i.p's wifi and the babel protocol by flooding it with
mcast, as the local mcast queue could easily grow 16+ seconds long.

um, I'm giving a preso tomorrow and will run behind this thread. It's
nice to see the renewed enthusiasm here, keep it up.

>> If there are no other stations competing for airtime, why does it matter that we use two txops?
>
> One further argument would be power consumption.  Radio transmitters eat batteries for lunch; the only consistently worse offender I can think of is a display backlight, assuming the software is efficient.

>  - Jonathan Morton
>
> _______________________________________________
> Bloat mailing list
> Bloat@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat



-- 

Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel: 1-669-226-2619

Re: [Make-wifi-fast] [Bloat] lwn.net's tcp small queues vs wifi aggregation solved

[Simon Barber]

Current versions of Wireshark have an experimental feature I added to 
expose airtime usage per packet and show 802.11 pcaps on a timeline.

Enable it under Preferences->Protocol->802.11 Radio

Simon

On June 25, 2018 6:27:59 PM Dave Taht <dave.taht@gmail.com> wrote:

> On Mon, Jun 25, 2018 at 5:44 PM, Jonathan Morton <chromatix99@gmail.com> wrote:
>>> On 26 Jun, 2018, at 3:36 am, Simon Barber <simon@superduper.net> wrote:
>>>
>>> Most hardware needs the packet finalized before it starts to contend for 
>>> the medium (as far as I’m aware - let me know if you know differently). One 
>>> issue is that if RTS/CTS is in use, then the packet duration needs to be 
>>> known in advance (or at least mid point of the RTS transmission).
>>
>> This is a valid argument.  I think we could successfully argue for a delay 
>> of 1ms, if there isn't already enough data in the queue to fill an 
>> aggregate, after the oldest packet arrives until a request is issued.
>
> Whoa, nelly! In the context of the local tcp stack over wifi, I was
> making an observation that I "frequently" saw a pattern of a single
> ack txop followed by a bunch in a separate txop. and I suggested a
> very short (10us) timeout before committing to the hw - not 1ms.
>
> Aside from this anecdote we have not got real data or statistics. The
> closest thing I have to a tool that can take apart wireless aircaps is
> here: https://github.com/dtaht/airtime-pie-chart which can be hacked
> to take more things apart than it currently does. Looking for this
> pattern in more traffic would be revealing in multiple ways. Looking
> for more patterns in bigger wifi networks would be good also.
>
> I like erics suggestion of doing more ack compression higher up in the
> tcp stack.
>
> There are two other things I've suggested in the past we look at. 1)
> The current fq_codel_for_wifi code has a philosophy of "one aggregate
> in the hardware, one ready to go". A simpler modification to fit more
> in would be to (wait the best case estimate for delivering the one in
> the hardware - a bit), then form the one ready-to-go.
>
> 2) rate limiting mcast and smoothing mcast bursts over time, allowing
> more unicast through. presently the mcast queue is infinite and very
> bursty. 802.11 std actually suggests mcast be rate limited by htb,
> where I'd be htb + fq + merging dup packets. I was routinely able to
> blow up the c.h.i.p's wifi and the babel protocol by flooding it with
> mcast, as the local mcast queue could easily grow 16+ seconds long.
>
> um, I'm giving a preso tomorrow and will run behind this thread. It's
> nice to see the renewed enthusiasm here, keep it up.
>
>>> If there are no other stations competing for airtime, why does it matter 
>>> that we use two txops?
>>
>> One further argument would be power consumption.  Radio transmitters eat 
>> batteries for lunch; the only consistently worse offender I can think of is 
>> a display backlight, assuming the software is efficient.
>
>>  - Jonathan Morton
>>
>> _______________________________________________
>> Bloat mailing list
>> Bloat@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/bloat
>
>
>
> --
>
> Dave Täht
> CEO, TekLibre, LLC
> http://www.teklibre.com
> Tel: 1-669-226-2619


Sent with AquaMail for Android
https://www.mobisystems.com/aqua-mail