From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-qk0-x22c.google.com (mail-qk0-x22c.google.com [IPv6:2607:f8b0:400d:c09::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 1C1583CB3A; Fri, 24 Aug 2018 14:43:14 -0400 (EDT) Received: by mail-qk0-x22c.google.com with SMTP id d15-v6so6497112qkc.1; Fri, 24 Aug 2018 11:43:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=odftRwbIlxYomOTEE75hfA7o05bSGX+rRUxmGUXAbZs=; b=OSqNIA+ibdekgmXEkTm7SQz8pkAA4nsCj/Vhzq8S69kzlProJ9mHUfvDGKOL5+AUKx Y7tm84JFCvSyLEdOCO0pQocC/lcB1zDSFw7FakB1u4jxzNSnyqKInLzWM1Ohq+eq33F+ sPOfzvEv0qW+pRgKZ6MouaFWjV5RamRuuYjutChcuCeG3tN6R5UF1jK1hmRrJPB+b/E8 iwNPIHXXn0nFxZOpEBTN/4D7AXBG/RTEsZaBOd5+k4GZbihCmVsN0fDh1N2OzqqlwM9z jVSvg8Jt7TZcdrn6ESr6ZFJhMLSxjawtfHDVwitZRfjvPTry6sIKZDh3dk1hB/BbE4d6 VfzQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=odftRwbIlxYomOTEE75hfA7o05bSGX+rRUxmGUXAbZs=; b=BMbPo75l9ZflN2QuDBCvWXSwFtPhef5I2csl8SivDKzuG4yx/DohkY6oWDq82VQxHM IcUItEnqJ8Dv/AplVFhF5Nkob3Z+d5EomnJANpc33zo75pesEr2enYHO0viJIcc/ptvc up2e3pUCcOsz/+pUm/qtS42o3BWwjUbkjsINiOlw0qYFzFXbiahtfQF9G1LLgwZ+Ih9E lpegJgBrpREs0+I1eR90AYAmERXTv/ZjYNopubOvYZd7k6i82gIhqztbUmGwOzEck1ty hLTwHFG7v4dMH7uqLBGDKjumThd5Pzte9DrMGw7taNYwq0kOKqxWoEUG/kMnCoMKTeGB O0sg== X-Gm-Message-State: APzg51BT2/nGimlof44rPcsrbLg1gVMajlkH3ZV+VVniPXLz75vbusWx Axnk5izujK3jqfDnSkrhA5Kdnnmhp4cm3m0OY03YzcJg X-Google-Smtp-Source: ANB0VdZTahUhpw8rtlwpF7CPCnGzbkATsxtPGRDVgkTUvTvG9njXXoo+0oid+fAJK76mrtNc0LFFDZvGiNbtUfg4nV0= X-Received: by 2002:a37:4f4b:: with SMTP id d72-v6mr2948893qkb.17.1535136193637; Fri, 24 Aug 2018 11:43:13 -0700 (PDT) MIME-Version: 1.0 References: <999D19F8-7E27-47CB-BC14-06193CC40943@heistp.net> <8E4244B7-C0AE-43D6-980F-C2370E662845@heistp.net> In-Reply-To: From: Dave Taht Date: Fri, 24 Aug 2018 11:43:01 -0700 Message-ID: To: Pete Heist Cc: bloat , cerowrt-devel@lists.bufferbloat.net Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Subject: Re: [Bloat] beating the drum for BQL X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Aug 2018 18:43:14 -0000 On Fri, Aug 24, 2018 at 10:30 AM Dave Taht wrote: > > On Fri, Aug 24, 2018 at 10:13 AM Pete Heist wrote: > > > > > > On Aug 23, 2018, at 10:26 AM, Pete Heist wrote: > > > > On Aug 23, 2018, at 2:49 AM, Dave Taht wrote: > > > > I had a chance to give a talk at broadcom recently, slides here: > > > > http://flent-fremont.bufferbloat.net/~d/broadcom_aug9.pdf > > > > > > Thanks for sharing, this is really useful, raising awareness where it m= atters. Quite a bit of content... :) > > > > Ubiquiti needs some work getting this into more of their products (Edge= MAX in particular). A good time to lobby for this might be, well a couple m= onths ago, as they=E2=80=99re producing alpha builds for their upcoming 2.0= release with kernel 4.9 and new Cavium/Mediatek/Octeon SDKs. I just asked = about the status in the EdgeRouter Beta forum, in case it finds the right e= yes before the release: > > > > https://community.ubnt.com/t5/EdgeRouter-Beta/BQL-support/m-p/2466657 > > > > > > This started a discussion, and no, so far it looks like there=E2=80=99s= no BQL support in the upcoming 2.0 release. > > > > For my own benefit, re-reading the original patch series comment (https= ://lwn.net/Articles/469652/) makes it sound like BQL is useful even without= AQM (original benchmarks were done with straight pfifo_fast). I didn=E2=80= =99t realize this, actually. If anything incorrect about BQL was said in th= is discussion, correct us, please=E2=80=A6 :) > > yes, bql is very useful even with pfifo fast. without BQL I doubt the > internet would be scaling as it is today in the dc, or on the smaller > hosts and devices that support it. It's in the mvneta, it's in the > ar71xx, with documented results there that I could dig up. (tho: > things like tsq are helping and mask the problem on simple tests) The > experiment I documented on the slides that kicked off this thread and > the other experiment on the systemd bug, easily show the benefit on > hosts forwarding packets (be they from local applications, coming from > various sources like docker containers, etc), and anyone can show what > goes wrong if you disable BQL nowadays, basically restoring linux-3.3 > behavior, with a very simple test: > > For I in /sys/class/net/your_device/queues/tx*/byte_queue_limits/limit_mi= n > do > echo 10000000 > $I > done > > so long as you run enough kinds of flows that don't engage TSQ. > > However, in the edgerouter w/offloads case all that part of the stack > has been short circuited into the offload engine. I don't know how > much buffering is in there on the new firmware, I'd done a few tests > on it in the old days, showing it to be around 10ms at gigE but even > that memory is kind of vague (the easy test here is slam two ports > into one), and for all I know the new firmware is worse, without going > back to track this new release. (I do have a few edgerouters but they > are all in production) > > There was also a paper on BQL a few years back that I can dig up.... The only academic analysis so of BQL i knew of was this: "bufferbloat systemic analysis": http://200 dot 131 dot 219 dot 61/publications/2014/its2014_bb.pdf - note that bufferbloat.net's filters don't let me post numeric urls and you can find the paywalled versions by searching for that title on google scholar. Or on sci-hub. I found that again by re-reading my preso to sigcomm 2014 "The value of repeatable experiments and negative results" - https://conferences.sigcomm.org/sigcomm/2014/doc/slides/137.pdf ) which - in addition to providing some value-able history and links to the bufferbloat, fq, and aqm efforts, is really one of my best rants *ever* aimed at the academic research and publication process. I enjoyed writing that, and giving the preso *a lot*. For some reason or another sigcomm has not invited me back. :) > > > > > Pete > > > > > -- > > Dave T=C3=A4ht > CEO, TekLibre, LLC > http://www.teklibre.com > Tel: 1-669-226-2619 --=20 Dave T=C3=A4ht CEO, TekLibre, LLC http://www.teklibre.com Tel: 1-669-226-2619