From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-qk1-x741.google.com (mail-qk1-x741.google.com [IPv6:2607:f8b0:4864:20::741]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 3CBC13BA8E for ; Tue, 15 Jan 2019 14:22:27 -0500 (EST) Received: by mail-qk1-x741.google.com with SMTP id 68so2245461qke.9 for ; Tue, 15 Jan 2019 11:22:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=UcQocIcV3xgKkm2oTU95J3vFVSs2kogC9Fwk6QU3s5M=; b=E2qLfz2z1d6Cylum/kbcj8Upc1RZftxo0PMvh9ntoMQL3Q+D/deg4670SnLGHpz5f4 gYnSMQTQsf3RN97FnzCCA2zOQa11/KNeT77T8zH9LZMNX4iJ4FyLiliZIHcp4l75eOF4 wsNkCG2ytk+FiQP7TERckfx5MCsjrsvoVCDRy0L6ELSPC5JjNeJ3aRVNOp/NW1owxg08 kGS1fgMQ66JlUD9oob8KjvZYLmfnH/HzsfnS+r9dB0WB2aConWFwScwgdP3zk2XVurkL 9kPGypFV9uToecokSVqOu0la6l75LNz5UCKen82B4e41PG51g9xnyUlxOuNJa5o6cnTY w9tQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=UcQocIcV3xgKkm2oTU95J3vFVSs2kogC9Fwk6QU3s5M=; b=fXkgZx0F9Ugq5sYfKHKCxEhnA5zvZ8+VZ1jwQorNSn0UoZ5nqiSpjn6fDEzunJGHpF 3khl1y3nIG+7L8Z1z1Ukli2joKO3Q3OnEvSQ1yq9deaj2sa/pG4PNHzc4v9y/6cXnEKs ohU1jYLQAd7ct7pHCcmZfRQzDoS35I4IransnSckzsYoncIfCFfXwRWqdKvDFtr8wDGl PxFycliLKGjFlbLXcfGTsZxp3BkVza087eGVyYYoPFmcisFeAK+ckY1++gxnGSpyeq2E nc65G+w6j4Kq5JYhaNrNJItWT6/fC/TWShi04gaVirAJvH+Iqeot1olNfdL6LEJlLQeN +Y9w== X-Gm-Message-State: AJcUukfTMETmymO/Iu7TWcB1hd+64rehilxN6CesBgQVXnqOzobTEpyZ SCW3GvoTI5MOXExMyUa0WaLvO3KOo3U= X-Google-Smtp-Source: ALg8bN49dWA9c2YkHGTHUzPnlNlBQMx6BwbEqzF8qI4nH/+MVJobivGw9wbtOu61ODhv2p/Fbqg9mw== X-Received: by 2002:a37:62d3:: with SMTP id w202mr3816976qkb.357.1547580146347; Tue, 15 Jan 2019 11:22:26 -0800 (PST) Received: from vmroute.lan (c-76-114-187-97.hsd1.md.comcast.net. [76.114.187.97]) by smtp.gmail.com with ESMTPSA id r5sm43401738qke.33.2019.01.15.11.22.24 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 15 Jan 2019 11:22:25 -0800 (PST) From: George Amanakis To: cake@lists.bufferbloat.net Date: Tue, 15 Jan 2019 14:22:00 -0500 Message-Id: <20190115192200.5644-1-gamanakis@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Subject: [Cake] dual-src/dsthost unfairness, only with bi-directional traffic X-BeenThere: cake@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Cake - FQ_codel the next generation List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Jan 2019 19:22:27 -0000 I think what is happening here is that if a client has flows such as "a (bulk upload)" and "b (bulk download)", the incoming ACKs of flow "a" compete with the incoming bulk traffic on flow "b". With compete I mean in terms of flow selection. So if we adjust the host_load to be the same with the bulk_flow_count of *each* host, the problem seems to be resolved. I drafted a patch below. Pete's setup, tested with the patch (ingress in mbit/s): IP1: 8down 49.18mbit/s IP1: 1up 46.73mbit/s IP2: 1down 47.39mbit/s IP2: 8up 49.21mbit/s --- sch_cake.c | 34 ++++++++++++++++++++++++++++------ 1 file changed, 28 insertions(+), 6 deletions(-) diff --git a/sch_cake.c b/sch_cake.c index d434ae0..5c0f0e1 100644 --- a/sch_cake.c +++ b/sch_cake.c @@ -148,6 +148,7 @@ struct cake_host { u32 dsthost_tag; u16 srchost_refcnt; u16 dsthost_refcnt; + u16 bulk_flow_count; }; struct cake_heap_entry { @@ -1897,10 +1898,10 @@ static s32 cake_enqueue(struct sk_buff *skb, struct Qdisc *sch, q->last_packet_time = now; } + struct cake_host *srchost = &b->hosts[flow->srchost]; + struct cake_host *dsthost = &b->hosts[flow->dsthost]; /* flowchain */ if (!flow->set || flow->set == CAKE_SET_DECAYING) { - struct cake_host *srchost = &b->hosts[flow->srchost]; - struct cake_host *dsthost = &b->hosts[flow->dsthost]; u16 host_load = 1; if (!flow->set) { @@ -1927,6 +1928,11 @@ static s32 cake_enqueue(struct sk_buff *skb, struct Qdisc *sch, flow->set = CAKE_SET_BULK; b->sparse_flow_count--; b->bulk_flow_count++; + if (cake_dsrc(q->flow_mode)) + srchost->bulk_flow_count++; + + if (cake_ddst(q->flow_mode)) + dsthost->bulk_flow_count++; } if (q->buffer_used > q->buffer_max_used) @@ -2101,7 +2107,7 @@ retry: host_load = max(host_load, srchost->srchost_refcnt); if (cake_ddst(q->flow_mode)) - host_load = max(host_load, dsthost->dsthost_refcnt); + host_load = max(host_load, dsthost->bulk_flow_count); WARN_ON(host_load > CAKE_QUEUES); @@ -2110,8 +2116,6 @@ retry: /* The shifted prandom_u32() is a way to apply dithering to * avoid accumulating roundoff errors */ - flow->deficit += (b->flow_quantum * quantum_div[host_load] + - (prandom_u32() >> 16)) >> 16; list_move_tail(&flow->flowchain, &b->old_flows); /* Keep all flows with deficits out of the sparse and decaying @@ -2122,6 +2126,11 @@ retry: if (flow->head) { b->sparse_flow_count--; b->bulk_flow_count++; + if (cake_dsrc(q->flow_mode)) + srchost->bulk_flow_count++; + + if (cake_ddst(q->flow_mode)) + dsthost->bulk_flow_count++; flow->set = CAKE_SET_BULK; } else { /* we've moved it to the bulk rotation for @@ -2131,6 +2140,8 @@ retry: flow->set = CAKE_SET_SPARSE_WAIT; } } + flow->deficit += (b->flow_quantum * quantum_div[host_load] + + (prandom_u32() >> 16)) >> 16; goto retry; } @@ -2151,6 +2162,11 @@ retry: &b->decaying_flows); if (flow->set == CAKE_SET_BULK) { b->bulk_flow_count--; + if (cake_dsrc(q->flow_mode)) + srchost->bulk_flow_count--; + + if (cake_ddst(q->flow_mode)) + dsthost->bulk_flow_count--; b->decaying_flow_count++; } else if (flow->set == CAKE_SET_SPARSE || flow->set == CAKE_SET_SPARSE_WAIT) { @@ -2164,8 +2180,14 @@ retry: if (flow->set == CAKE_SET_SPARSE || flow->set == CAKE_SET_SPARSE_WAIT) b->sparse_flow_count--; - else if (flow->set == CAKE_SET_BULK) + else if (flow->set == CAKE_SET_BULK) { b->bulk_flow_count--; + if (cake_dsrc(q->flow_mode)) + srchost->bulk_flow_count--; + + if (cake_ddst(q->flow_mode)) + dsthost->bulk_flow_count--; + } else b->decaying_flow_count--; -- 2.20.1