[Make-wifi-fast] Experimental ath10k airtime fairness patch?

[Make-wifi-fast] Experimental ath10k airtime fairness patch?

[Noah Causin]

Hi,

I was wondering if there is an experimental airtime fairness patch for 
the ath10k on LEDE.

I have been using Toke's patch for enabling intermediate queue support 
on the ath10k, and it has been working well.

Noah Causin

Re: [Make-wifi-fast] Experimental ath10k airtime fairness patch?

[Michal Kazior]

[-- Attachment #1: Type: text/plain, Size: 945 bytes --]

On 14 November 2016 at 18:06, Noah Causin <n0manletter@gmail.com> wrote:
> Hi,
>
> I was wondering if there is an experimental airtime fairness patch for the
> ath10k on LEDE.
>
> I have been using Toke's patch for enabling intermediate queue support on
> the ath10k, and it has been working well.

I did some work on that but didn't finish nor publish it.

I just dug around my git repo and found something. I can't remember if
it was this exactly what I used in the "100 station" thingy[1][2] or
not. Attached.

Feel free to play around it if you want. It should provide fairness
but will degrade peak performance due to delayed firmware aggregation
introducing extra idling latency and the naive design.

You'll most likely need to rebase it.

[1]: http://imgur.com/a/sSn8V
[2]: http://www.linuxplumbersconf.com/2016/ocw//system/presentations/3963/original/linuxplumbers_wifi_latency-3Nov.pdf
page 28, 30


Michał

[-- Attachment #2: 0001-ath10k-introduce-per-sta-DQL-and-tx-scheduling.patch --]
[-- Type: text/x-diff, Size: 11004 bytes --]

From 1486ffbafdb4dd433203c35c694b7443fa769210 Mon Sep 17 00:00:00 2001
From: Michal Kazior <michal.kazior@tieto.com>
Date: Thu, 5 May 2016 13:41:18 +0200
Subject: [PATCH] ath10k: introduce per-sta DQL and tx scheduling

Work in progress.

Signed-off-by: Michal Kazior <michal.kazior@tieto.com>
---
 drivers/net/wireless/ath/ath10k/core.c   |  3 +-
 drivers/net/wireless/ath/ath10k/core.h   |  9 ++-
 drivers/net/wireless/ath/ath10k/htt_rx.c | 21 ++++++-
 drivers/net/wireless/ath/ath10k/htt_tx.c |  9 ++-
 drivers/net/wireless/ath/ath10k/mac.c    | 97 +++++++++++++++++++++++---------
 drivers/net/wireless/ath/ath10k/txrx.c   |  9 ++-
 drivers/net/wireless/ath/ath10k/txrx.h   |  3 +-
 7 files changed, 116 insertions(+), 35 deletions(-)

diff --git a/drivers/net/wireless/ath/ath10k/core.c b/drivers/net/wireless/ath/ath10k/core.c
index e94cb87380d2..31d34852586b 100644
--- a/drivers/net/wireless/ath/ath10k/core.c
+++ b/drivers/net/wireless/ath/ath10k/core.c
@@ -2100,7 +2100,8 @@ struct ath10k *ath10k_core_create(size_t priv_size, struct device *dev,
 	spin_lock_init(&ar->data_lock);
 	spin_lock_init(&ar->txqs_lock);
 
-	INIT_LIST_HEAD(&ar->txqs);
+	INIT_LIST_HEAD(&ar->txqs_new);
+	INIT_LIST_HEAD(&ar->txqs_old);
 	INIT_LIST_HEAD(&ar->peers);
 	init_waitqueue_head(&ar->peer_mapping_wq);
 	init_waitqueue_head(&ar->htt.empty_tx_wq);
diff --git a/drivers/net/wireless/ath/ath10k/core.h b/drivers/net/wireless/ath/ath10k/core.h
index 1379054000f9..683d1d050106 100644
--- a/drivers/net/wireless/ath/ath10k/core.h
+++ b/drivers/net/wireless/ath/ath10k/core.h
@@ -310,8 +310,13 @@ struct ath10k_peer {
 
 struct ath10k_txq {
 	struct list_head list;
+	struct list_head completed_list;
+	struct dql dql;
+	ktime_t tx_start;
 	unsigned long num_fw_queued;
 	unsigned long num_push_allowed;
+	unsigned int completed; /* protected by: htt->tx_lock */
+	s64 deficit;
 };
 
 struct ath10k_sta {
@@ -830,7 +835,9 @@ struct ath10k {
 	/* protects: ar->txqs, artxq->list */
 	spinlock_t txqs_lock;
 
-	struct list_head txqs;
+	struct ath10k_txq *txq;
+	struct list_head txqs_new;
+	struct list_head txqs_old;
 	struct list_head arvifs;
 	struct list_head peers;
 	struct ath10k_peer *peer_map[ATH10K_MAX_NUM_PEER_IDS];
diff --git a/drivers/net/wireless/ath/ath10k/htt_rx.c b/drivers/net/wireless/ath/ath10k/htt_rx.c
index 7411b69fde42..806bdc6d3fdf 100644
--- a/drivers/net/wireless/ath/ath10k/htt_rx.c
+++ b/drivers/net/wireless/ath/ath10k/htt_rx.c
@@ -1635,7 +1635,7 @@ static void ath10k_htt_rx_tx_compl_ind(struct ath10k *ar,
 		if (!kfifo_put(&htt->txdone_fifo, tx_done)) {
 			ath10k_warn(ar, "txdone fifo overrun, msdu_id %d status %d\n",
 				    tx_done.msdu_id, tx_done.status);
-			ath10k_txrx_tx_unref(htt, &tx_done);
+			ath10k_txrx_tx_unref(htt, &tx_done, NULL);
 		}
 	}
 }
@@ -2285,7 +2285,7 @@ bool ath10k_htt_t2h_msg_handler(struct ath10k *ar, struct sk_buff *skb)
 			break;
 		}
 
-		status = ath10k_txrx_tx_unref(htt, &tx_done);
+		status = ath10k_txrx_tx_unref(htt, &tx_done, NULL);
 		if (!status) {
 			spin_lock_bh(&htt->tx_lock);
 			ath10k_htt_tx_mgmt_dec_pending(htt);
@@ -2410,10 +2410,12 @@ static void ath10k_htt_txrx_compl_task(unsigned long ptr)
 {
 	struct ath10k_htt *htt = (struct ath10k_htt *)ptr;
 	struct ath10k *ar = htt->ar;
+	struct ath10k_txq *artxq;
 	struct htt_tx_done tx_done = {};
 	struct sk_buff_head rx_ind_q;
 	struct sk_buff_head tx_ind_q;
 	struct sk_buff *skb;
+	struct list_head completed_list;
 	unsigned long flags;
 	int num_mpdus;
 
@@ -2428,13 +2430,26 @@ static void ath10k_htt_txrx_compl_task(unsigned long ptr)
 	skb_queue_splice_init(&htt->tx_fetch_ind_q, &tx_ind_q);
 	spin_unlock_irqrestore(&htt->tx_fetch_ind_q.lock, flags);
 
+	INIT_LIST_HEAD(&completed_list);
+
 	/* kfifo_get: called only within txrx_tasklet so it's neatly serialized.
 	 * From kfifo_get() documentation:
 	 *  Note that with only one concurrent reader and one concurrent writer,
 	 *  you don't need extra locking to use these macro.
 	 */
 	while (kfifo_get(&htt->txdone_fifo, &tx_done))
-		ath10k_txrx_tx_unref(htt, &tx_done);
+		ath10k_txrx_tx_unref(htt, &tx_done, &completed_list);
+
+	while (!list_empty(&completed_list)) {
+		artxq = list_first_entry(&completed_list, struct ath10k_txq,
+					 completed_list);
+		list_del_init(&artxq->completed_list);
+
+		spin_lock_bh(&htt->tx_lock);
+		dql_completed(&artxq->dql, artxq->completed);
+		artxq->completed = 0;
+		spin_unlock_bh(&htt->tx_lock);
+	}
 
 	while ((skb = __skb_dequeue(&tx_ind_q))) {
 		ath10k_htt_rx_tx_fetch_ind(ar, skb);
diff --git a/drivers/net/wireless/ath/ath10k/htt_tx.c b/drivers/net/wireless/ath/ath10k/htt_tx.c
index 6269c610b0a3..668f03e7d914 100644
--- a/drivers/net/wireless/ath/ath10k/htt_tx.c
+++ b/drivers/net/wireless/ath/ath10k/htt_tx.c
@@ -379,7 +379,7 @@ static int ath10k_htt_tx_clean_up_pending(int msdu_id, void *skb, void *ctx)
 	tx_done.msdu_id = msdu_id;
 	tx_done.status = HTT_TX_COMPL_STATE_DISCARD;
 
-	ath10k_txrx_tx_unref(htt, &tx_done);
+	ath10k_txrx_tx_unref(htt, &tx_done, NULL);
 
 	return 0;
 }
@@ -841,12 +841,14 @@ int ath10k_htt_tx(struct ath10k_htt *htt, enum ath10k_hw_txrx_mode txmode,
 	struct ieee80211_hdr *hdr = (struct ieee80211_hdr *)msdu->data;
 	struct ieee80211_tx_info *info = IEEE80211_SKB_CB(msdu);
 	struct ath10k_skb_cb *skb_cb = ATH10K_SKB_CB(msdu);
+	struct ath10k_txq *artxq = (void *)skb_cb->txq->drv_priv;
 	struct ath10k_hif_sg_item sg_items[2];
 	struct ath10k_htt_txbuf *txbuf;
 	struct htt_data_tx_desc_frag *frags;
 	bool is_eth = (txmode == ATH10K_HW_TXRX_ETHERNET);
 	u8 vdev_id = ath10k_htt_tx_get_vdev_id(ar, msdu);
 	u8 tid = ath10k_htt_tx_get_tid(msdu, is_eth);
+	size_t msdu_len;
 	int prefetch_len;
 	int res;
 	u8 flags0 = 0;
@@ -1015,12 +1017,17 @@ int ath10k_htt_tx(struct ath10k_htt *htt, enum ath10k_hw_txrx_mode txmode,
 	sg_items[1].paddr = skb_cb->paddr;
 	sg_items[1].len = prefetch_len;
 
+	msdu_len = msdu->len;
+
 	res = ath10k_hif_tx_sg(htt->ar,
 			       htt->ar->htc.endpoint[htt->eid].ul_pipe_id,
 			       sg_items, ARRAY_SIZE(sg_items));
 	if (res)
 		goto err_unmap_msdu;
 
+	if (skb_cb->txq)
+		dql_queued(&artxq->dql, msdu_len);
+
 	return 0;
 
 err_unmap_msdu:
diff --git a/drivers/net/wireless/ath/ath10k/mac.c b/drivers/net/wireless/ath/ath10k/mac.c
index 1bae547a9616..708c63e3fd66 100644
--- a/drivers/net/wireless/ath/ath10k/mac.c
+++ b/drivers/net/wireless/ath/ath10k/mac.c
@@ -3652,6 +3652,8 @@ static void ath10k_mac_txq_init(struct ieee80211_txq *txq)
 		return;
 
 	INIT_LIST_HEAD(&artxq->list);
+	INIT_LIST_HEAD(&artxq->completed_list);
+	dql_init(&artxq->dql, HZ);
 }
 
 static void ath10k_mac_txq_unref(struct ath10k *ar, struct ieee80211_txq *txq)
@@ -3776,40 +3778,81 @@ void ath10k_mac_tx_push_pending(struct ath10k *ar)
 {
 	struct ieee80211_hw *hw = ar->hw;
 	struct ieee80211_txq *txq;
+	struct list_head *list;
 	struct ath10k_txq *artxq;
-	struct ath10k_txq *last;
+	ktime_t now;
+	u64 sent;
+	s64 delta;
 	int ret;
-	int max;
-
-	if (ar->htt.num_pending_tx >= (ar->htt.max_num_pending_tx / 2))
-		return;
 
 	spin_lock_bh(&ar->txqs_lock);
 	rcu_read_lock();
 
-	last = list_last_entry(&ar->txqs, struct ath10k_txq, list);
-	while (!list_empty(&ar->txqs)) {
-		artxq = list_first_entry(&ar->txqs, struct ath10k_txq, list);
-		txq = container_of((void *)artxq, struct ieee80211_txq,
-				   drv_priv);
-
-		/* Prevent aggressive sta/tid taking over tx queue */
-		max = 16;
-		ret = 0;
-		while (ath10k_mac_tx_can_push(hw, txq) && max--) {
-			ret = ath10k_mac_tx_push_txq(hw, txq);
-			if (ret < 0)
+	for (;;) {
+		list = &ar->txqs_new;
+		if (list_empty(list)) {
+			list = &ar->txqs_old;
+			if (list_empty(list))
 				break;
 		}
 
-		list_del_init(&artxq->list);
-		if (ret != -ENOENT)
-			list_add_tail(&artxq->list, &ar->txqs);
-
-		ath10k_htt_tx_txq_update(hw, txq);
-
-		if (artxq == last || (ret < 0 && ret != -ENOENT))
+		artxq = list_first_entry_or_null(list, struct ath10k_txq, list);
+		if (!artxq)
 			break;
+
+		if (artxq->deficit <= 0) {
+			artxq->deficit += 300;
+			list_move_tail(&artxq->list, &ar->txqs_old);
+			continue;
+		}
+
+		if (ar->txq)
+			artxq = ar->txq;
+
+		txq = container_of((void *)artxq, struct ieee80211_txq, drv_priv);
+		ret = 0;
+		sent = 0;
+
+		if (ar->txq != artxq) {
+			artxq->tx_start = ktime_get_raw();
+
+			for (;;) {
+				if (dql_avail(&artxq->dql) < 0)
+					break;
+
+				if (!ath10k_mac_tx_can_push(hw, txq))
+					break;
+
+				ret = ath10k_mac_tx_push_txq(hw, txq);
+				if (ret < 0)
+					break;
+
+				sent += ret;
+			}
+
+			ath10k_htt_tx_txq_update(hw, txq);
+
+			if (ret == -ENOENT && !sent) {
+				list_del_init(&artxq->list);
+				continue;
+			}
+
+			ar->txq = artxq;
+		}
+
+		if (artxq->num_fw_queued == 0) {
+			now = ktime_get_raw();
+			delta = ktime_us_delta(now, artxq->tx_start);
+			artxq->tx_start = now;
+			artxq->deficit -= delta;
+			ar->txq = NULL;
+			printk("txq %pM delta %llu usec\n",
+					txq->sta ? txq->sta->addr : txq->vif->addr,
+					delta);
+		}
+
+		list_move_tail(&artxq->list, &ar->txqs_old);
+		break;
 	}
 
 	rcu_read_unlock();
@@ -4050,8 +4093,10 @@ static void ath10k_mac_op_wake_tx_queue(struct ieee80211_hw *hw,
 	struct ath10k_txq *artxq = (void *)txq->drv_priv;
 
 	spin_lock_bh(&ar->txqs_lock);
-	if (list_empty(&artxq->list))
-		list_add_tail(&artxq->list, &ar->txqs);
+	if (list_empty(&artxq->list)) {
+		list_add_tail(&artxq->list, &ar->txqs_new);
+		artxq->deficit = 300;
+	}
 	spin_unlock_bh(&ar->txqs_lock);
 
 	ath10k_mac_tx_push_pending(ar);
diff --git a/drivers/net/wireless/ath/ath10k/txrx.c b/drivers/net/wireless/ath/ath10k/txrx.c
index 1966c787998b..746644b2483f 100644
--- a/drivers/net/wireless/ath/ath10k/txrx.c
+++ b/drivers/net/wireless/ath/ath10k/txrx.c
@@ -50,7 +50,8 @@ static void ath10k_report_offchan_tx(struct ath10k *ar, struct sk_buff *skb)
 }
 
 int ath10k_txrx_tx_unref(struct ath10k_htt *htt,
-			 const struct htt_tx_done *tx_done)
+			 const struct htt_tx_done *tx_done,
+			 struct list_head *completed_list)
 {
 	struct ath10k *ar = htt->ar;
 	struct device *dev = ar->dev;
@@ -83,8 +84,12 @@ int ath10k_txrx_tx_unref(struct ath10k_htt *htt,
 	txq = skb_cb->txq;
 	artxq = (void *)txq->drv_priv;
 
-	if (txq)
+	if (txq) {
 		artxq->num_fw_queued--;
+		artxq->completed += msdu->len;
+		if (completed_list && list_empty(&artxq->completed_list))
+			list_add_tail(&artxq->completed_list, completed_list);
+	}
 
 	ath10k_htt_tx_free_msdu_id(htt, tx_done->msdu_id);
 	ath10k_htt_tx_dec_pending(htt);
diff --git a/drivers/net/wireless/ath/ath10k/txrx.h b/drivers/net/wireless/ath/ath10k/txrx.h
index e7ea1ae1c438..0fe5e6fcf7e5 100644
--- a/drivers/net/wireless/ath/ath10k/txrx.h
+++ b/drivers/net/wireless/ath/ath10k/txrx.h
@@ -20,7 +20,8 @@
 #include "htt.h"
 
 int ath10k_txrx_tx_unref(struct ath10k_htt *htt,
-			 const struct htt_tx_done *tx_done);
+			 const struct htt_tx_done *tx_done,
+			 struct list_head *completed_list);
 
 struct ath10k_peer *ath10k_peer_find(struct ath10k *ar, int vdev_id,
 				     const u8 *addr);
-- 
2.1.4

Re: [Make-wifi-fast] Experimental ath10k airtime fairness patch?

[Noah Causin]

Thank you for the patch.


On 11/15/2016 1:36 PM, Michal Kazior wrote:
> On 14 November 2016 at 18:06, Noah Causin <n0manletter@gmail.com> wrote:
>> Hi,
>>
>> I was wondering if there is an experimental airtime fairness patch for the
>> ath10k on LEDE.
>>
>> I have been using Toke's patch for enabling intermediate queue support on
>> the ath10k, and it has been working well.
> I did some work on that but didn't finish nor publish it.
>
> I just dug around my git repo and found something. I can't remember if
> it was this exactly what I used in the "100 station" thingy[1][2] or
> not. Attached.
>
> Feel free to play around it if you want. It should provide fairness
> but will degrade peak performance due to delayed firmware aggregation
> introducing extra idling latency and the naive design.
>
> You'll most likely need to rebase it.
>
> [1]: http://imgur.com/a/sSn8V
> [2]: http://www.linuxplumbersconf.com/2016/ocw//system/presentations/3963/original/linuxplumbers_wifi_latency-3Nov.pdf
> page 28, 30
>
>
> Michał

Re: [Make-wifi-fast] Experimental ath10k airtime fairness patch?

[Valent Turkovic]

[-- Attachment #1: Type: text/plain, Size: 1623 bytes --]

Hi guys,
this news is awesome! Aritime fairness is really missing feature.

Has anyone done some tests? What kind of difference do you see with and
without the patch? Has anyone tested the patch with 20+ clients?

Cheers,
Valent.

On Thu, Nov 17, 2016 at 12:59 AM, Noah Causin <n0manletter@gmail.com> wrote:

> Thank you for the patch.
>
>
>
> On 11/15/2016 1:36 PM, Michal Kazior wrote:
>
>> On 14 November 2016 at 18:06, Noah Causin <n0manletter@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I was wondering if there is an experimental airtime fairness patch for
>>> the
>>> ath10k on LEDE.
>>>
>>> I have been using Toke's patch for enabling intermediate queue support on
>>> the ath10k, and it has been working well.
>>>
>> I did some work on that but didn't finish nor publish it.
>>
>> I just dug around my git repo and found something. I can't remember if
>> it was this exactly what I used in the "100 station" thingy[1][2] or
>> not. Attached.
>>
>> Feel free to play around it if you want. It should provide fairness
>> but will degrade peak performance due to delayed firmware aggregation
>> introducing extra idling latency and the naive design.
>>
>> You'll most likely need to rebase it.
>>
>> [1]: http://imgur.com/a/sSn8V
>> [2]: http://www.linuxplumbersconf.com/2016/ocw//system/presentati
>> ons/3963/original/linuxplumbers_wifi_latency-3Nov.pdf
>> page 28, 30
>>
>>
>> Michał
>>
>
> _______________________________________________
> Make-wifi-fast mailing list
> Make-wifi-fast@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/make-wifi-fast
>

[-- Attachment #2: Type: text/html, Size: 2855 bytes --]

Re: [Make-wifi-fast] Experimental ath10k airtime fairness patch?

[Dave Taht]

With versions of the patchset going back to august, we tested airtime
fairness with 4, 12, and 30 under a variety of conditions. It looked
*really good*. In one scenario, with one slow station and 28 "faster"
ones, we got 5x more throughput through the AP while retaining low
latency.

I'd planned, til recently, to roll it out to my production (babel)
mesh at the yurtlab this month, but various factors (like having to
climb roofs in the winter, and some other bugs in powersave, recently
resolved, and a nagging issue with TSQ), have thus far stopped me.
There have been so many other mods elsewhere in the stack since that
original round of testing that a full on set of tests with lede head
are needed, and certainly my hope is a few more teams get excited
enough to do evaluations in their environments.

On Thu, Nov 17, 2016 at 1:04 AM, Valent Turkovic
<valent@otvorenamreza.org> wrote:
> Hi guys,
> this news is awesome! Aritime fairness is really missing feature.
>
> Has anyone done some tests? What kind of difference do you see with and
> without the patch? Has anyone tested the patch with 20+ clients?
>
> Cheers,
> Valent.
>
> On Thu, Nov 17, 2016 at 12:59 AM, Noah Causin <n0manletter@gmail.com> wrote:
>>
>> Thank you for the patch.
>>
>>
>>
>> On 11/15/2016 1:36 PM, Michal Kazior wrote:
>>>
>>> On 14 November 2016 at 18:06, Noah Causin <n0manletter@gmail.com> wrote:
>>>>
>>>> Hi,
>>>>
>>>> I was wondering if there is an experimental airtime fairness patch for
>>>> the
>>>> ath10k on LEDE.
>>>>
>>>> I have been using Toke's patch for enabling intermediate queue support
>>>> on
>>>> the ath10k, and it has been working well.
>>>
>>> I did some work on that but didn't finish nor publish it.
>>>
>>> I just dug around my git repo and found something. I can't remember if
>>> it was this exactly what I used in the "100 station" thingy[1][2] or
>>> not. Attached.
>>>
>>> Feel free to play around it if you want. It should provide fairness
>>> but will degrade peak performance due to delayed firmware aggregation
>>> introducing extra idling latency and the naive design.
>>>
>>> You'll most likely need to rebase it.
>>>
>>> [1]: http://imgur.com/a/sSn8V
>>> [2]:
>>> http://www.linuxplumbersconf.com/2016/ocw//system/presentations/3963/original/linuxplumbers_wifi_latency-3Nov.pdf
>>> page 28, 30
>>>
>>>
>>> Michał
>>
>>
>> _______________________________________________
>> Make-wifi-fast mailing list
>> Make-wifi-fast@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/make-wifi-fast
>
>
>
> _______________________________________________
> Make-wifi-fast mailing list
> Make-wifi-fast@lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/make-wifi-fast
>



-- 
Dave Täht
Let's go make home routers and wifi faster! With better software!
http://blog.cerowrt.org

Re: [Make-wifi-fast] Experimental ath10k airtime fairness patch?

[Toke Høiland-Jørgensen]

"Jesse (Jie) Zhang" <JJZhang@SonicWALL.com> writes:

> Hi,
> Ath10k  support airtime fairness ? where can I get this pathch.

Not yet. Most of the required infrastructure is there, but there is
still a little way to get this supported... :)

-Toke

Re: [Make-wifi-fast] Experimental ath10k airtime fairness patch?

[Jesse (Jie) Zhang]

[-- Attachment #1: Type: text/plain, Size: 78 bytes --]

Hi,
Ath10k  support airtime fairness ? where can I get this pathch.
Thanks

[-- Attachment #2: Type: text/html, Size: 2115 bytes --]