<div dir="auto">I am going to give it a try, with your patch applied tonight and report.<div dir="auto">Thank you!</div><div dir="auto"><br></div><div dir="auto">George </div></div><br><div class="gmail_quote"><div dir="ltr">On Thu, Jul 5, 2018, 6:31 PM Toke Høiland-Jørgensen <<a href="mailto:toke@toke.dk">toke@toke.dk</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Toke Høiland-Jørgensen <<a href="mailto:toke@toke.dk" target="_blank" rel="noreferrer">toke@toke.dk</a>> writes:<br>
<br>
> Jonathan Morton <<a href="mailto:chromatix99@gmail.com" target="_blank" rel="noreferrer">chromatix99@gmail.com</a>> writes:<br>
><br>
>>> On 3 Jul, 2018, at 1:23 am, Toke Høiland-Jørgensen <<a href="mailto:toke@toke.dk" target="_blank" rel="noreferrer">toke@toke.dk</a>> wrote:<br>
>>> <br>
>>> My hunch is that this has something to do with the way mlx5 uses<br>
>>> multiple receive queues (and thus multiple CPUs). Which is probably<br>
>>> different from veth...<br>
>><br>
>> At this stage I'm pretty confident it has nothing to do with Cake, and<br>
>> everything to do with the Mellanox hardware and driver. It does strike<br>
>> me that Linux' default handling of multiqueue hardware doesn't map<br>
>> very well to the qdisc interface.<br>
><br>
> Well, it doesn't happen with fq_codel, so even if it is a driver bug, it<br>
> is being triggered by cake specifically...<br>
<br>
Right, so finally got some time to investigate this further.<br>
<br>
I suspected that cake_dequeue() was looping forever, so I added some<br>
debug statements to investigate this; and turns out I was right. Using<br>
the debug patch below, in unlimited mode I get loop aborts on loop 'i'<br>
for unlimited mode and loop 'l' if I enable the shaper at 70 gbit. It<br>
happens pretty reliably, but only when I load up the link sufficiently<br>
(need 4-6 TCP flows which get ~50 Gbps of total throughput).<br>
<br>
The weird thing is that what appears to be happening, is that cake<br>
somehow gets into a state where sch->q.qlen is >0 while all tin backlogs<br>
are 0. I have no clue how this happens; as far as I can tell, all<br>
changes to tin_backlog are paired with a change to q.qlen. The only<br>
thing outside of cake itself that modifies q.qlen is peek(), which is<br>
not being used here.<br>
<br>
I'm giving up for tonight; if anyone else has any ideas, I'm all ears.<br>
<br>
-Toke<br>
<br>
Sample debug output:<br>
<br>
[ 5456.068281] Loop counter i hit 100k; aborting! i 100001 j 0 k 180 l 3 m 0 qlen 2 qbkllog 33184 tin 2 deficit 172 tot backlog 0<br>
<br>
With this debug patch:<br>
<br>
@@ -1892,6 +1892,20 @@ static struct sk_buff *cake_dequeue(struct Qdisc *sch)<br>
u64 delay;<br>
u32 len;<br>
<br>
+ int i=0,j=0,k=0,l=0,m=0;<br>
+<br>
+#define COUNT_LOOP(v) do { \<br>
+ if (++v > 100000) { \<br>
+ int tot_bkl = 0; \<br>
+ struct cake_tin_data *t; \<br>
+ int n; \<br>
+ for(n=0,t = q->tins; n < CAKE_MAX_TINS; n++,t++) \<br>
+ tot_bkl += t->tin_backlog; \<br>
+ net_warn_ratelimited("Loop counter " #v " hit 100k; aborting! i %d j %d k %d l %d m %d qlen %d qbkllog %d tin %d deficit %d tot backlog %d", i, j, k, l, m, sch->q.qlen, sch->qstats.backlog, q->cur_tin, b->tin_deficit, tot_bkl); \<br>
+ return NULL; \<br>
+ } \<br>
+ } while(0);<br>
+<br>
begin:<br>
if (!sch->q.qlen)<br>
return NULL;<br>
@@ -1912,6 +1926,7 @@ begin:<br>
/* In unlimited mode, can't rely on shaper timings, just balance<br>
* with DRR<br>
*/<br>
+ i=0;<br>
while (b->tin_deficit < 0 ||<br>
!(b->sparse_flow_count + b->bulk_flow_count)) {<br>
if (b->tin_deficit <= 0)<br>
@@ -1923,6 +1938,7 @@ begin:<br>
q->cur_tin = 0;<br>
b = q->tins;<br>
}<br>
+ COUNT_LOOP(i);<br>
}<br>
} else {<br>
/* In shaped mode, choose:<br>
@@ -1960,8 +1976,10 @@ retry:<br>
head = &b->old_flows;<br>
if (unlikely(list_empty(head))) {<br>
head = &b->decaying_flows;<br>
- if (unlikely(list_empty(head)))<br>
+ if (unlikely(list_empty(head))) {<br>
+ COUNT_LOOP(j);<br>
goto begin;<br>
+ }<br>
}<br>
}<br>
}<br>
@@ -2008,6 +2026,7 @@ retry:<br>
flow->set = CAKE_SET_SPARSE_WAIT;<br>
}<br>
}<br>
+ COUNT_LOOP(k);<br>
goto retry;<br>
}<br>
<br>
@@ -2050,6 +2069,7 @@ retry:<br>
srchost->srchost_refcnt--;<br>
dsthost->dsthost_refcnt--;<br>
}<br>
+ COUNT_LOOP(l);<br>
goto begin;<br>
}<br>
<br>
@@ -2075,6 +2095,8 @@ retry:<br>
kfree_skb(skb);<br>
if (q->rate_flags & CAKE_FLAG_INGRESS)<br>
goto retry;<br>
+<br>
+ COUNT_LOOP(m);<br>
}<br>
<br>
b->tin_ecn_mark += !!flow->cvars.ecn_marked;<br>
<br>
<br>
<br>
</blockquote></div>