Re: [Make-wifi-fast] TCP performance regression in mac80211 triggered by the fq code

Re: [Make-wifi-fast] TCP performance regression in mac80211 triggered by the fq code

[Dave Taht]

On Tue, Jul 12, 2016 at 12:09 PM, Felix Fietkau <nbd@nbd.name> wrote:
> Hi,
>
> With Toke's ath9k txq patch I've noticed a pretty nasty performance
> regression when running local iperf on an AP (running the txq stuff) to
> a wireless client.

Your kernel? cpu architecture?

What happens when going through the AP to a server from the wireless client?

Which direction?

> Here's some things that I found:
> - when I use only one TCP stream I get around 90-110 Mbit/s

with how much cpu left over?

> - when running multiple TCP streams, I get only 35-40 Mbit/s total

with how much cpu left over?
context switch difference between the two tests?
tcp_limit_output_bytes is?

got perf?

> - fairness between TCP streams looks completely fine

A codel will get to long term fairness pretty fast. Packet captures
from a fq will show much more regular interleaving of packets,
regardless.

> - there's no big queue buildup, the code never actually drops any packets

A "trick" I have been using to observe codel behavior has been to
enable ecn on server and client, then checking in wireshark for ect(3)
marked packets.

> - if I put a hack in the fq code to force the hash to a constant value

You could also set "flows" to 1 to keep the hash being generated, but
not actually use it.

> (effectively disabling fq without disabling codel), the problem
> disappears and even multiple streams get proper performance.

Meaning you get 90-110Mbits ?

Do you have a "before toke" figure for this platform?

> Please let me know if you have any ideas.

I am in berlin, packing hardware...

>
> - Felix
> --
> To unsubscribe from this list: send the line "unsubscribe linux-wireless" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Dave Täht
Let's go make home routers and wifi faster! With better software!
http://blog.cerowrt.org

Re: [Make-wifi-fast] TCP performance regression in mac80211 triggered by the fq code

[Felix Fietkau]

On 2016-07-12 14:13, Dave Taht wrote:
> On Tue, Jul 12, 2016 at 12:09 PM, Felix Fietkau <nbd@nbd.name> wrote:
>> Hi,
>>
>> With Toke's ath9k txq patch I've noticed a pretty nasty performance
>> regression when running local iperf on an AP (running the txq stuff) to
>> a wireless client.
> 
> Your kernel? cpu architecture?
QCA9558, 720 MHz, running Linux 4.4.14

> What happens when going through the AP to a server from the wireless client?
Will test that next.

> Which direction?
AP->STA, iperf running on the AP. Client is a regular MacBook Pro
(Broadcom).

>> Here's some things that I found:
>> - when I use only one TCP stream I get around 90-110 Mbit/s
> 
> with how much cpu left over?
~20%

>> - when running multiple TCP streams, I get only 35-40 Mbit/s total
> with how much cpu left over?
~30%

> context switch difference between the two tests?
What's the easiest way to track that?

> tcp_limit_output_bytes is?
262144

> got perf?
Need to make a new build for that.

>> - fairness between TCP streams looks completely fine
> 
> A codel will get to long term fairness pretty fast. Packet captures
> from a fq will show much more regular interleaving of packets,
> regardless.
> 
>> - there's no big queue buildup, the code never actually drops any packets
> 
> A "trick" I have been using to observe codel behavior has been to
> enable ecn on server and client, then checking in wireshark for ect(3)
> marked packets.
I verified this with printk. The same issue already appears if I have
just the fq patch (with the codel patch reverted).

>> - if I put a hack in the fq code to force the hash to a constant value
> 
> You could also set "flows" to 1 to keep the hash being generated, but
> not actually use it.
> 
>> (effectively disabling fq without disabling codel), the problem
>> disappears and even multiple streams get proper performance.
> 
> Meaning you get 90-110Mbits ?
Right.

> Do you have a "before toke" figure for this platform?
It's quite similar.

>> Please let me know if you have any ideas.
> 
> I am in berlin, packing hardware...
Nice!

- Felix

Re: [Make-wifi-fast] TCP performance regression in mac80211 triggered by the fq code

[Dave Taht]

On Tue, Jul 12, 2016 at 3:21 PM, Felix Fietkau <nbd@nbd.name> wrote:
> On 2016-07-12 14:13, Dave Taht wrote:
>> On Tue, Jul 12, 2016 at 12:09 PM, Felix Fietkau <nbd@nbd.name> wrote:
>>> Hi,
>>>
>>> With Toke's ath9k txq patch I've noticed a pretty nasty performance
>>> regression when running local iperf on an AP (running the txq stuff) to
>>> a wireless client.
>>
>> Your kernel? cpu architecture?
> QCA9558, 720 MHz, running Linux 4.4.14
>
>> What happens when going through the AP to a server from the wireless client?
> Will test that next.
>
>> Which direction?
> AP->STA, iperf running on the AP. Client is a regular MacBook Pro
> (Broadcom).

There are always 2 wifi chips in play. Like the Sith.

>>> Here's some things that I found:
>>> - when I use only one TCP stream I get around 90-110 Mbit/s
>>
>> with how much cpu left over?
> ~20%
>
>>> - when running multiple TCP streams, I get only 35-40 Mbit/s total
>> with how much cpu left over?
> ~30%

Hmm.

Care to try netperf?

>
>> context switch difference between the two tests?
> What's the easiest way to track that?

if you have gnu "time" time -v the_process

or:

perf record -e context-switches -ag

or: process /proc/$PID/status for cntx

>> tcp_limit_output_bytes is?
> 262144

I keep hoping to be able to reduce this to something saner like 4096
one day. It got bumped to 64k based on bad wifi performance once, and
then to it's current size to make the Xen folk happier.

The other param I'd like to see fiddled with is tcp_notsent_lowat.

In both cases reductions will increase your context switches but
reduce memory pressure and lead to a more reactive tcp.

And in neither case I think this is the real cause of this problem.


>> got perf?
> Need to make a new build for that.
>
>>> - fairness between TCP streams looks completely fine
>>
>> A codel will get to long term fairness pretty fast. Packet captures
>> from a fq will show much more regular interleaving of packets,
>> regardless.
>>
>>> - there's no big queue buildup, the code never actually drops any packets
>>
>> A "trick" I have been using to observe codel behavior has been to
>> enable ecn on server and client, then checking in wireshark for ect(3)
>> marked packets.
> I verified this with printk. The same issue already appears if I have
> just the fq patch (with the codel patch reverted).

OK. A four flow test "should" trigger codel....

Running out of cpu (or hitting some other bottleneck), without
loss/marking "should" result in a tcptrace -G and xplot.org of the
packet capture showing the window continuing to increase....


>>> - if I put a hack in the fq code to force the hash to a constant value
>>
>> You could also set "flows" to 1 to keep the hash being generated, but
>> not actually use it.
>>
>>> (effectively disabling fq without disabling codel), the problem
>>> disappears and even multiple streams get proper performance.
>>
>> Meaning you get 90-110Mbits ?
> Right.
>
>> Do you have a "before toke" figure for this platform?
> It's quite similar.
>
>>> Please let me know if you have any ideas.
>>
>> I am in berlin, packing hardware...
> Nice!
>
> - Felix
>



-- 
Dave Täht
Let's go make home routers and wifi faster! With better software!
http://blog.cerowrt.org

Re: [Make-wifi-fast] TCP performance regression in mac80211 triggered by the fq code

[Dave Taht]

On Tue, Jul 12, 2016 at 4:02 PM, Dave Taht <dave.taht@gmail.com> wrote:
> On Tue, Jul 12, 2016 at 3:21 PM, Felix Fietkau <nbd@nbd.name> wrote:
>> On 2016-07-12 14:13, Dave Taht wrote:
>>> On Tue, Jul 12, 2016 at 12:09 PM, Felix Fietkau <nbd@nbd.name> wrote:
>>>> Hi,
>>>>
>>>> With Toke's ath9k txq patch I've noticed a pretty nasty performance
>>>> regression when running local iperf on an AP (running the txq stuff) to
>>>> a wireless client.
>>>
>>> Your kernel? cpu architecture?
>> QCA9558, 720 MHz, running Linux 4.4.14

So this is a single core at the near-bottom end of the range. I guess
we also should find a MIPS 24c derivative that runs at 400Mhz or so.

What HZ? (I no longer know how much higher HZ settings make any
difference, but I'm usually at NOHZ and 250, rather than 100.)

And all the testing to date was on much higher end multi-cores.

>>> What happens when going through the AP to a server from the wireless client?
>> Will test that next.

Anddddd?

>>
>>> Which direction?
>> AP->STA, iperf running on the AP. Client is a regular MacBook Pro
>> (Broadcom).
>
> There are always 2 wifi chips in play. Like the Sith.
>
>>>> Here's some things that I found:
>>>> - when I use only one TCP stream I get around 90-110 Mbit/s
>>>
>>> with how much cpu left over?
>> ~20%
>>
>>>> - when running multiple TCP streams, I get only 35-40 Mbit/s total
>>> with how much cpu left over?
>> ~30%

To me this implies a contending lock issue, too much work in the irq
handler or too delayed work in the softirq handler....

I thought you were very brave to try and backport this.

>
> Hmm.
>
> Care to try netperf?
>
>>
>>> context switch difference between the two tests?
>> What's the easiest way to track that?
>
> if you have gnu "time" time -v the_process
>
> or:
>
> perf record -e context-switches -ag
>
> or: process /proc/$PID/status for cntx
>
>>> tcp_limit_output_bytes is?
>> 262144
>
> I keep hoping to be able to reduce this to something saner like 4096
> one day. It got bumped to 64k based on bad wifi performance once, and
> then to it's current size to make the Xen folk happier.
>
> The other param I'd like to see fiddled with is tcp_notsent_lowat.
>
> In both cases reductions will increase your context switches but
> reduce memory pressure and lead to a more reactive tcp.
>
> And in neither case I think this is the real cause of this problem.
>
>
>>> got perf?
>> Need to make a new build for that.
>>
>>>> - fairness between TCP streams looks completely fine
>>>
>>> A codel will get to long term fairness pretty fast. Packet captures
>>> from a fq will show much more regular interleaving of packets,
>>> regardless.
>>>
>>>> - there's no big queue buildup, the code never actually drops any packets
>>>
>>> A "trick" I have been using to observe codel behavior has been to
>>> enable ecn on server and client, then checking in wireshark for ect(3)
>>> marked packets.
>> I verified this with printk. The same issue already appears if I have
>> just the fq patch (with the codel patch reverted).
>
> OK. A four flow test "should" trigger codel....
>
> Running out of cpu (or hitting some other bottleneck), without
> loss/marking "should" result in a tcptrace -G and xplot.org of the
> packet capture showing the window continuing to increase....
>
>
>>>> - if I put a hack in the fq code to force the hash to a constant value
>>>
>>> You could also set "flows" to 1 to keep the hash being generated, but
>>> not actually use it.
>>>
>>>> (effectively disabling fq without disabling codel), the problem
>>>> disappears and even multiple streams get proper performance.
>>>
>>> Meaning you get 90-110Mbits ?
>> Right.
>>
>>> Do you have a "before toke" figure for this platform?
>> It's quite similar.
>>
>>>> Please let me know if you have any ideas.
>>>
>>> I am in berlin, packing hardware...
>> Nice!
>>
>> - Felix
>>
>
>
>
> --
> Dave Täht
> Let's go make home routers and wifi faster! With better software!
> http://blog.cerowrt.org



-- 
Dave Täht
Let's go make home routers and wifi faster! With better software!
http://blog.cerowrt.org

Re: [Make-wifi-fast] TCP performance regression in mac80211 triggered by the fq code

[Felix Fietkau]

On 2016-07-13 09:57, Dave Taht wrote:
> On Tue, Jul 12, 2016 at 4:02 PM, Dave Taht <dave.taht@gmail.com> wrote:
>> On Tue, Jul 12, 2016 at 3:21 PM, Felix Fietkau <nbd@nbd.name> wrote:
>>> On 2016-07-12 14:13, Dave Taht wrote:
>>>> On Tue, Jul 12, 2016 at 12:09 PM, Felix Fietkau <nbd@nbd.name> wrote:
>>>>> Hi,
>>>>>
>>>>> With Toke's ath9k txq patch I've noticed a pretty nasty performance
>>>>> regression when running local iperf on an AP (running the txq stuff) to
>>>>> a wireless client.
>>>>
>>>> Your kernel? cpu architecture?
>>> QCA9558, 720 MHz, running Linux 4.4.14
> 
> So this is a single core at the near-bottom end of the range. I guess
> we also should find a MIPS 24c derivative that runs at 400Mhz or so.
> 
> What HZ? (I no longer know how much higher HZ settings make any
> difference, but I'm usually at NOHZ and 250, rather than 100.)
> 
> And all the testing to date was on much higher end multi-cores.
> 
>>>> What happens when going through the AP to a server from the wireless client?
>>> Will test that next.
> 
> Anddddd?
Single stream: 130 Mbit/s, 70% idle
Two streams: 50-80 Mbit/s (wildly fluctuating), 73% idle.

>>>> Which direction?
>>> AP->STA, iperf running on the AP. Client is a regular MacBook Pro
>>> (Broadcom).
>>
>> There are always 2 wifi chips in play. Like the Sith.
>>
>>>>> Here's some things that I found:
>>>>> - when I use only one TCP stream I get around 90-110 Mbit/s
>>>>
>>>> with how much cpu left over?
>>> ~20%
>>>
>>>>> - when running multiple TCP streams, I get only 35-40 Mbit/s total
>>>> with how much cpu left over?
>>> ~30%
> 
> To me this implies a contending lock issue, too much work in the irq
> handler or too delayed work in the softirq handler....
> 
> I thought you were very brave to try and backport this.
I don't think this has anything to do with contending locks, CPU
utilization, etc. The code does something to the packets that TCP really
doesn't like.

- Felix

Re: [Make-wifi-fast] TCP performance regression in mac80211 triggered by the fq code

[Dave Taht]

On Wed, Jul 13, 2016 at 10:53 AM, Felix Fietkau <nbd@nbd.name> wrote:

>> To me this implies a contending lock issue, too much work in the irq
>> handler or too delayed work in the softirq handler....
>>
>> I thought you were very brave to try and backport this.
> I don't think this has anything to do with contending locks, CPU
> utilization, etc. The code does something to the packets that TCP really
> doesn't like.

With your 70% idle figure, I am inclined to agree... could you get an aircap
of the two different tests?  - as well as a regular packetcap taken at
the client or server?
And put somewhere I can get at them?

What version of OSX are you running?

I will setup an ath9k box shortly...


-- 
Dave Täht
Let's go make home routers and wifi faster! With better software!
http://blog.cerowrt.org

Re: [Make-wifi-fast] TCP performance regression in mac80211 triggered by the fq code

[Michal Kazior]

On 12 July 2016 at 16:02, Dave Taht <dave.taht@gmail.com> wrote:
[...]
>>> tcp_limit_output_bytes is?
>> 262144
>
> I keep hoping to be able to reduce this to something saner like 4096
> one day. It got bumped to 64k based on bad wifi performance once, and
> then to it's current size to make the Xen folk happier.

Not sure if it's possible. You do need this to be at least as big as a
single A-MPDU can get. In extreme 11ac cases it can be pretty big.

I recall a discussion from a long time ago and the tcp limit output
logic was/is coupled with the assumption that tx-completions always
come max 1ms after tx submission. This is rather tricky to *guarantee*
on wifi, especially with firmware blobs, big aggregates, lots of
stations and retries.


Michał

Re: [Make-wifi-fast] TCP performance regression in mac80211 triggered by the fq code

[Dave Taht]

On Tue, Jul 12, 2016 at 2:57 PM, Toke Høiland-Jørgensen <toke@toke.dk> wrote:
> Dave Taht <dave.taht@gmail.com> writes:
>
>>> As for why this would happen... There could be a bug in the dequeue code
>>> somewhere, but since you get better performance from sticking everything
>>> into one queue, my best guess would be that the client is choking on the
>>> interleaved packets? I.e. expending more CPU when it can't stick
>>> subsequent packets into the same TCP flow?
>>
>> I share this concern.
>>
>> The quantum is? I am not opposed to a larger quantum (2 full size
>> packets = 3028 in this case?).
>
> The quantum is hard-coded to 300 bytes in the current implementation
> (see net/fq_impl.h).

don't do that. :)

A single full size packet is preferable, and saves going around the
main dequeue loop 5-6 times per flow on this workload.

My tests on the prior patch set were mostly at the larger quantum.


> -Toke



-- 
Dave Täht
Let's go make home routers and wifi faster! With better software!
http://blog.cerowrt.org