Re: [Cerowrt-devel] performance numbers from WRT1200AC (Re: Latest build test - new sqm-scripts seem to work; "cake overhead 40" didn't)

[Sebastian Moeller <moeller0@gmx.de>]

[-- Attachment #1: Type: text/plain, Size: 3875 bytes --]

Hi List,

On Jun 29, 2015, at 18:44 , Dave Taht <dave.taht@gmail.com> wrote:

> On Mon, Jun 29, 2015 at 6:42 AM, Sebastian Moeller <moeller0@gmx.de> wrote:
>> HI Mikael,, hi Jonathan,
>> 
>>> [...]
>>> 
>>> These are the results from 50M and 500M, also including 50up and 50down that I added to my test suite script.
>>> 
>>> http://swm.pp.se/aqm/rrul_150629-cake-4.tar
>>> 
>> 
>> Now both ingress and egress are up to roughly 455Mbps from roughly 360 with cake just playing leaf qdisc for HTB. This looks even better than before…
> 
> 350 *usec* induced delay on the 50mbit rrul_be test. w00t!
> 
> Most of the tests compare well with the reference rangeley data now. I
> would like a 900mbit soft shaped result.

	Make sure to use the correct per packet overhead of 18 + 20, on gigabit ethernet inter-frame gap and preamble cost 20 Bytes worth of data. So with a MTU of 1500 thee is no issue (900 * (1538/1518) = 911.85770751 Mbps) but the smaller the packets get:

900 * (MTU+38/MTU+18) = 1000
(MTU+38/MTU+18) = (1000 / 900)
MTU+38 = (10 / 9) * (MTU + 18)
MTU + 38 = (10/9) * MTU + (10/9)*18
38 - (10/9)*18 = (10/9) * MTU - (9/9)MTU
38 - (10/9)*18 = (1/9) MTU
MTU = (38 - ((10/9)*18))*9  = 162
So for TCP/IPv4 MSS < 122 the shaper will not keep the ethernet hardware queues empty… 

On the other hand shaping at 
1000/(88/64) = 727.272727273 Mbps
should make sure that even at minimal packet size of 64byte shaping would still be keeping the ethernet queues “empty-ish”. If the 1200ac can shape at 900 I would rather specify the correct overhead though.


To make things a bit trickier, depending on the interface used the kernel will already account for the standards ethernet header without the frame check sequence, so I would guess in the 900Mbps soft shaper on ethN scenario one would need to add a per packet overhead of 24 bytes. If someone in the know could double check that reasoning I would be much obliged…


Best Regards
	Sebastian



> 
> 1.2ms at 500mbit. Less of a w00t. Possible it is coming from elsewhere
> on that path (fq or fq_codel on the server and client?)
> 
> cake currently peels at 1ms / flows (more or less)... NAPI is an
> issue... hw mq an issue...
> 
> There are a half dozen things in the mvneta driver I would try to
> reduce it's latency more. The easy ones:
> 
> reduce this to 16:
> 
>        netif_napi_add(dev, &pp->napi, mvneta_poll, NAPI_POLL_WEIGHT);
> 
> Reduce this to 24: (this will also reduce the max outstanding stuff in
> the driver by  a LOT, but is still not BQL!)
> 
> /* Max number of allowed TCP segments for software TSO */
> #define MVNETA_MAX_TSO_SEGS 100
> 
> Both of the will improve read side latency at the cost of more sirqs.
> 
> I do not know what reducing these will do, and would test both of the
> above separately.
> 
> /* Coalescing */
> #define MVNETA_TXDONE_COAL_PKTS         1
> #define MVNETA_RX_COAL_PKTS             32
> #define MVNETA_RX_COAL_USEC             100
> 
> As for cake itself, eric dumazet told us we dont need atomic ops in it,
> and peeling at at even lower threshold has some appeal (to me, anyway)
> 
> attached is a patch for that, put it in your feeds/cero/kmod_sched_cake/patches
> directory, rebuild (make package/kmod-sched-cake/{clean,compile,install})
> 
> (bump up the makefile rel number also, if you want)
> 
> 
> 
> 
> 
>> Best Regards
>>        Sebastian
>> _______________________________________________
>> Cerowrt-devel mailing list
>> Cerowrt-devel@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
> 
> 
> 
> -- 
> Dave Täht
> worldwide bufferbloat report:
> http://www.dslreports.com/speedtest/results/bufferbloat
> And:
> What will it take to vastly improve wifi for everyone?
> https://plus.google.com/u/0/explore/makewififast

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Rid-unneeded-atomic-ops-and-reduce-peeling-threshold.patch --]
[-- Type: text/x-patch; name="0001-Rid-unneeded-atomic-ops-and-reduce-peeling-threshold.patch", Size: 2580 bytes --]

From 46be609e95474e9db856b5e12756d4a7568adf42 Mon Sep 17 00:00:00 2001
From: Dave Taht <dave.taht@bufferbloat.net>
Date: Mon, 29 Jun 2015 09:38:00 -0700
Subject: [PATCH] Rid unneeded atomic ops and reduce peeling threshold

---
 sch_cake.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/sch_cake.c b/sch_cake.c
index 80e1cb2..9a358b9 100644
--- a/sch_cake.c
+++ b/sch_cake.c
@@ -121,7 +121,7 @@ struct cake_fqcd_sched_data {
 
     struct codel_params cparams;
     u32      drop_overlimit;
-    atomic_t flow_count;
+    u32 flow_count;
 
     struct list_head new_flows; /* list of new flows */
     struct list_head old_flows; /* list of old flows */
@@ -427,7 +427,7 @@ static int cake_enqueue(struct sk_buff *skb, struct Qdisc *sch)
 	 * Split GSO aggregates if they're likely to impair flow isolation
 	 * or if we need to know individual packet sizes for framing overhead.
 	 */
-	if(unlikely((len * max(atomic_read(&fqcd->flow_count), 1)) > q->peel_threshold && skb_is_gso(skb)))
+	if(unlikely((len * max(&fqcd->flow_count, 1)) > q->peel_threshold && skb_is_gso(skb)))
 	{
 		struct sk_buff *segs, *nskb;
 		netdev_features_t features = netif_skb_features(skb);
@@ -477,7 +477,7 @@ static int cake_enqueue(struct sk_buff *skb, struct Qdisc *sch)
 	/* flowchain */
 	if(list_empty(&flow->flowchain)) {
 		list_add_tail(&flow->flowchain, &fqcd->new_flows);
-		atomic_inc(&fqcd->flow_count);
+		fqcd->flow_count+=1;
 		flow->deficit = fqcd->quantum;
 		flow->dropped = 0;
 	}
@@ -615,7 +615,7 @@ retry:
 			list_move_tail(&flow->flowchain, &fqcd->old_flows);
 		} else {
 			list_del_init(&flow->flowchain);
-			atomic_dec(&fqcd->flow_count);
+			fqcd->flow_count-=1;
 		}
 		goto begin;
 	}
@@ -966,7 +966,7 @@ static void cake_reconfigure(struct Qdisc *sch)
 		if(q->buffer_limit < 65536)
 			q->buffer_limit = 65536;
 
-		q->peel_threshold = (q->rate_flags & CAKE_FLAG_ATM) ? 0 : min(65535U, q->rate_bps >> 10);
+		q->peel_threshold = (q->rate_flags & CAKE_FLAG_ATM) ? 0 : min(65535U, q->rate_bps >> 12);
 	} else {
 		q->buffer_limit = 1 << 20;
 		q->peel_threshold = 0;
@@ -1083,7 +1083,7 @@ static int cake_init(struct Qdisc *sch, struct nlattr *opt)
 		fqcd->perturbation = prandom_u32();
 		INIT_LIST_HEAD(&fqcd->new_flows);
 		INIT_LIST_HEAD(&fqcd->old_flows);
-		atomic_set(&fqcd->flow_count, 0);
+		fqcd->flow_count = 0;
 		/* codel_params_init(&fqcd->cparams); */
 
 		fqcd->flows    = cake_zalloc(fqcd->flows_cnt * sizeof(struct cake_fqcd_flow));
-- 
1.9.1


[-- Attachment #3: Type: text/plain, Size: 4 bytes --]

>