From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-yh0-f47.google.com (mail-yh0-f47.google.com [209.85.213.47]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by huchra.bufferbloat.net (Postfix) with ESMTPS id A94F021F4E4 for ; Thu, 4 Dec 2014 08:09:42 -0800 (PST) Received: by mail-yh0-f47.google.com with SMTP id f73so8367011yha.20 for ; Thu, 04 Dec 2014 08:09:42 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-type; bh=CHTIEV6Eaf90vBBM096B9+IAB1u8QT2DsiuhSJMS/ac=; b=LvjLoSVon7maNkFSNLvhXU5iK7m1UwwjCgLGcLv1WAKHBB7ZzdhDYizu1U5uOwW7es RDxyQszGV961hpwuuJLQWKN3AlWDQkyy7ddsqiS1vlkTTYQQ1VxaYDPOQEU8VPUxfJRp rsNwNs+9TfOfmJ2O0z+0b9JPRJWtFIXetcp6oEN1/78N/Us5/VTQ8XMcisigDO4NRQew 3l8Q6k+rOAKK80qIsjzL/jijFoohZEM/Opn7PrqFI7j52eRw4Io10gDArq/iA9Hdv/VA 93RDbmL4bqFhz9hcN7ErL7huqgj1TA+WrytQOtRE92lf7tOdz/FpgF500mAeIfuzCdv7 Ywvg== X-Gm-Message-State: ALoCoQnD1LXb97t8q17qKaHvMYh02O1fejfZuXRrIXB3ZdmeD5FdElJ6RbWfAXYSPRVb8R2VanjI X-Received: by 10.170.204.23 with SMTP id v23mr15665868yke.115.1417709381826; Thu, 04 Dec 2014 08:09:41 -0800 (PST) MIME-Version: 1.0 Received: by 10.170.145.195 with HTTP; Thu, 4 Dec 2014 08:09:21 -0800 (PST) X-Originating-IP: [2a02:fe0:c120:1350:7e7a:91ff:fe0e:3e2c] In-Reply-To: References: <121767.1413574248@turing-police.cc.vt.edu> <9382.1413826910@turing-police.cc.vt.edu> <544672D3.8020709@redhat.com> <58702.1413908858@turing-police.cc.vt.edu> <54469242.5010506@redhat.com> <54469B96.50204@redhat.com> From: Tom Gundersen Date: Thu, 4 Dec 2014 17:09:21 +0100 Message-ID: To: Dave Taht Content-Type: text/plain; charset=UTF-8 Cc: "cerowrt-devel@lists.bufferbloat.net" Subject: Re: [Cerowrt-devel] SQM in mainline openwrt, fq_codel considered for fedora default X-BeenThere: cerowrt-devel@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: Development issues regarding the cerowrt test router project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Dec 2014 16:10:12 -0000 On Tue, Oct 21, 2014 at 10:59 PM, Dave Taht wrote: > On Tue, Oct 21, 2014 at 12:51 PM, Dave Taht wrote: >>> http://snapon.lab.bufferbloat.net/~d/beagle_bql/bql_makes_a_difference.png >>> >>> You can see that BQL makes the most difference in the latency. >> >> And ALSO that these fixes also improved system throughput enormously. > > Meant to include that plot. > > http://snapon.lab.bufferbloat.net/~d/beagle_bql/beaglebonewithbql.png > > You can disregard the decline in download bandwidth (as we are also > sending 5x as many acks and measurement data, which is not counted > in that part of the plot) > >> This is partially due to the improvement in ack clocking you get from >> reduced RTTs, partially due to improved cache behavior (shorter >> queues), and partially continual improvements elsewhere in the tcp >> portions of the stack. >> >> With more recent kernels... >> >> I now get full throughput from the beagles in both directions with the >> 3.16 kernel, >> the stil out of tree bql patch, and either fq or fq_codel. I haven't >> got around to plotting all those results (they are from kathie's new >> lab), but they are here: >> http://snapon.lab.bufferbloat.net/~d/pollere/ > > The latency spikes are generally due to not having BQL, probably: > http://snapon.lab.bufferbloat.net/~d/pollere/beagle/beagle-3.8-nobql-fq-fq_codel.png > > This is using the new fq scheduler on both sides, with BQL enabled. > > http://snapon.lab.bufferbloat.net/~d/pollere/beagle/beagle_3.16-fq-fq-no-offloads.png > > The switch most likely is prioritizing EF marked packets. (as is > sch_fq). Most of the buffering is in the switch, not the host, now. > (the prior results I showed had no switch in the way) > >> There is a small buffered tail dropping switch in the way, on these >> later data sets. There was some puzzling behavior on the e1000e that I >> need to poke into in a more controlled setting. >> >> As for other tunables on hosts, TCP small queues might be amiable to >> some tuning, but that too may well evolve further in kernelspace. > > So I have now drowned you in data on one architecture and setup. The > most thoroughly publicly analyzed devices and drivers are the ar71xx, > e1000e, and beaglebone at this point. > > The use of fq_codel in a qos system (artificially rate limited using > htb, hfsc, or tbf) is pretty well proven to be a huge win at this > point. > > At line rates fq and fq_codel still help quite a bit without a BQL > enabled driver, BQL is needed for best results. I don't know to what > extent the BQL enabled drivers already cover the marketplace, it was > generally my assumption that the e1000e counted for a lot... > > http://www.bufferbloat.net/projects/bloat/wiki/BQL_enabled_drivers > > And thus with all the positive results so far, more wider distribution > of the new stuff > on more devices outside the sample set of those on the bloat, > cerowrt-devel and codel lists (about 500 people all told), > > and all ya gotta do is turn it on. Thanks for the very detailed info Dave. What I'm taking from this is that at the moment we should not be doing anything more than what we already do (the sysctl changes that Michal pushed), and that any improvements should be made in the kernel drivers (BQL support in particular). What we could do in the future, is in case you have kernel features that should be enabled by default as they benefit the general user, but where you cannot change the kernel defaults due to backwards compat, we can (as we did with fq) add these to the recommended sysctl files shipped with systemd. This is of course just a gentle nudge for people to use features, distros/admins can still override them. > This gets me to the stopping point that we hit a whlle back, which was > reliably determining if a good clocksource was present in the system. > Somehow. clock_getres, perhaps? I appear to have missed the context on this. In what situation do we (userspace) need to care about the existence of a high quality clocksource? Cheers, Tom