From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-qk0-x22e.google.com (mail-qk0-x22e.google.com [IPv6:2607:f8b0:400d:c09::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 355113B2A4 for ; Sun, 19 Aug 2018 21:32:27 -0400 (EDT) Received: by mail-qk0-x22e.google.com with SMTP id b19-v6so1003454qkc.6 for ; Sun, 19 Aug 2018 18:32:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to:cc :content-transfer-encoding; bh=rmdT++S79xJBln5IhwyHmV6uiN4j6Wg/L1Zgeypt2cI=; b=QYphNU1TPMUV6uDxIKBLST3Bifa8P1PzMTmNP2NME0u6NK+/82RgPmSwyWZDItM9vQ 3L/tIb8ogRbnAec/wvnwWOTdCn5dxLFO8MRNViEZoJUiEp6QhVU9MOxLU7NNP4MroWTX 5JT0zL/s7S8UKy68hIe3Qr15MbK0dCdjW/BgMkJehAMf3L+lRIoJeJi6upwxGMaej4Ct QJfKO+IyHPDBNJhyI+EP+RhsFSHcyGO2EX1LsAqgR3pVthVXLMQyedOLQfw+xaKMrC6c P9Lm1vPhCvWJgZf9vrOYtED2rf+3Q5S8WNjvwZNb4fwv+S9kaTAvAVD5J6wjw238ezry C+cw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to:cc :content-transfer-encoding; bh=rmdT++S79xJBln5IhwyHmV6uiN4j6Wg/L1Zgeypt2cI=; b=ah8fBd1ZOGD0bpiSUe3rIfffbbkHqTbis/4GvWrs6XYOsdndrBsDTIkq67lj5yYxY0 kTYmGkZ6/YJ6loIfCuqc/qMF4LmUbnHzUesrk5zTRwlG6ajRE5JQMNmBVN0pIZy4K+2c ihSCbEtlytKFjYQpMa6QZuDyFWv6EKylVTr9ddqjB9kHbBbGWoBst5XeB2D9KwBVEj3v Ym8kKphd7rbA5Q3dKUt+7A+m3v3vZr5E2XoztJvTxHqmD/405rqPjVdr1INhZ73tYf+w /NgH1rMo20zX9NTFuQK3jKKdHzcCJk8/7LgQeLpgEvHRmGqpb0VAEpMqpgZU4gZ7dvJc WLuw== X-Gm-Message-State: AOUpUlHs5Fl3YeEgl7qgpSiSEBoQrLANS+qHa87MrE8/dB/gggwWIwn1 OOzmuQwcHb5idIqMlZjpHSU2AjGtinOZjBkBCANTxQfl X-Google-Smtp-Source: AA+uWPyABm7ds4LM2LCsmvASY4gRBD4OoFSwobxWXjvCrVGKWoszXuGBaJJ9mldXKX4PbaXXmhLnPNRPbB7nT71dgcM= X-Received: by 2002:a37:12ce:: with SMTP id 75-v6mr6257009qks.319.1534728746309; Sun, 19 Aug 2018 18:32:26 -0700 (PDT) MIME-Version: 1.0 From: Dave Taht Date: Sun, 19 Aug 2018 18:32:12 -0700 Message-ID: To: codel@lists.bufferbloat.net Cc: Eric Dumazet Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Subject: [Codel] codel janitorial: increasing signal strength on serious overload, and tail drop, and other ideas X-BeenThere: codel@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: CoDel AQM discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Aug 2018 01:32:27 -0000 Based on the carnage over on this bug report: https://github.com/systemd/systemd/issues/9725 and after exploring various options (I really want the pacing rate to be independent of the cwnd, but don't know how)... I'm contemplating two ideas for when fq_codel has lost control of the queue, neither "solves" the problem stated there, but might help in the general case. There's been a few others too... 1) increase codel's signal strength on a bulk drop, by a bulk increment count and reschedule next drop. I *loved* the bulk dropper idea, as it made a big difference cpu-wise while responding to udp floods on a string of tests in an early deployment. https://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git/tree/net= /sched/sch_fq_codel.c#n153 However, I think (!) it should also basically replicate what codel would have done here: https://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git/tree/inc= lude/net/codel_impl.h#n176 to increase the drop rate for that flow. And that's easy, just slam in vars->count+=3Di; codel_Newton_step(vars); codel_control_law(now, params->interval, vars->rec_inv_sqrt); looks like a win to me. I'll go experiment. 2) add tail drop mode for local packets pfifo-fast and sch_fq will refuse additional packets when the packet limit is hit. The local stack, won't drop them, but reduce cwnd and resubmit. Codel is usually running below its limits in the first place, but if we identify that a packet is from the local stack (how?), perhaps we can just push back there instead of hitting the bulk dropper. I don't care for this idea because I feel it should hit the fattest flow instead, or be exerting backpressure via some other means. Neither of these ideas actually solve "the problem", and I didn't agree with either the problem statement or the proposed solution on that particular bug report either, and it took me a while to do a variety of experiments (and calm down). Other stuff that has been in my out of tree patchset for a while have been: 3) getting rid of the ce_threshold mode, which so far I know (?) was the topic of a failed experiment elsewhere in google and entirely unused anywhere else I know of. It's hard to delete "features", in linux, but I could put it out there (at least for openwrt) and see if anyone screams. Saves 32 bits and a couple if statements. 4) the flow->dropped stat is not very useful as it is reset every time the queue empties and is only visible if you do a tc class dump. It could be made useful by not zeroing it out, but I don't think it's used.... https://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git/tree/net= /sched/sch_fq_codel.c#n217 5) It seems possible to get the bulk dropper to be closer to O(1) by merely keeping track of the max backlogged flow continuously. You do a bulk drop, and then you have up to 64 rounds of fq to find the next biggest fat flow. Moving the per flow backlog statistic https://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git/tree/net= /sched/sch_fq_codel.c#n60 out of its own array and sticking it in codel_vars will save on a cache miss as well. There happens to be 15 spare bits in codel_vars (https://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git/tree/ne= t/sched/sch_fq_codel.c#n217) a maxbacklog of packet len shifted right 6 leaves 4MB for that which seems almost enough...... and it doesn't need to be that accurate as we keep track of the total backlog elsewhere. (part of my reason for proposing 3 & 4 is that makes 5 a zero cost new option, and after the cpu monster cake became I had an urge to save cycles somewhere to balance my karma) 6) I don't think the bulk dropper ever made it to the wifi code? --=20 Dave T=C3=A4ht CEO, TekLibre, LLC http://www.teklibre.com Tel: 1-669-226-2619