From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-qt0-x232.google.com (mail-qt0-x232.google.com [IPv6:2607:f8b0:400d:c0d::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 761DE3CB58; Fri, 24 Aug 2018 13:30:34 -0400 (EDT) Received: by mail-qt0-x232.google.com with SMTP id z8-v6so10964098qto.9; Fri, 24 Aug 2018 10:30:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=UObFYe64uMrJJr74FJjGYdK9+5NzU+tjnX5ioS2iV5Y=; b=jAjLYFMDgf90KnJrNrHpdny0gJWSMxmK1fKtgDpLlYlfA0xWmh/8RYTYF0ePip49ks rx1Eyw/snxjf7kXfbn+e4o98YQi9Fdzg0KfqBouOWjZ0cqXSZhJMfoV/xhK9OSPiXbC3 9jNsdhZP9sI3ylFAGFB4UEMF0+DkzUldsEjfv2uLexogSK8YLF+/hrYW8s8BrqrC1nbn zL/GpRiQRFnR2DCo+VCe/aJ1VPlObKcQJ2jUc1haGktdxxLmLb6jgUhtdPeFms5vqp55 FIzM0a8Kk2wcKMwqYM4mUPdfLWgutD6/XoTQweobdXif2Y8BkmYxGyujR+pB2pc9dxdB 9Rpw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=UObFYe64uMrJJr74FJjGYdK9+5NzU+tjnX5ioS2iV5Y=; b=FyMhiX+RM8dhC3TLSUBQgfcgNCB5TcOnFYgO1qEjA6ZUjJPYY2TDoP4QHFsf8d6Oek T0CvSAsGoZ8hQ+xws7S29+6Bgolb2hz2iEdrxmurcH7ELl2NbboYAjmgbO9leYQTYAvH AlpipPwnrTvbzr2CVq9e2yhjwDsp7+LHUg038MeieD2CcgttgkfSQkVcyDW3Ez2c9lWa utUcIoViCKJ3qy2TWTm5Qwp64KO2hO6GBSibETYPOhc0lRJsPukdENaC7bz4EHDESftz FnwQdaOIxgrN4/qNsvGGNOR5LoM/kADRvYMWUwVNHCVdGYIBb09mttWfNqGpXxXoEx8w v+tg== X-Gm-Message-State: APzg51Dij1syqwlLabyU6ofziTx8aaDL3hKToRBTN7JovnXlNNiH4Pdp Q9IsdHheJempxCRAZ1RgrE4hd18Z1jsfZCwxEpw= X-Google-Smtp-Source: ANB0Vdbz0npEdfamv1GGZFpOVeXDbV/OSyN3x7pw811p6R2g/9gNae7u8u8vk+KgArq/1nFs1ZjjFcoRpMvODwfecJY= X-Received: by 2002:a0c:818b:: with SMTP id 11-v6mr2877635qvd.129.1535131833893; Fri, 24 Aug 2018 10:30:33 -0700 (PDT) MIME-Version: 1.0 References: <999D19F8-7E27-47CB-BC14-06193CC40943@heistp.net> <8E4244B7-C0AE-43D6-980F-C2370E662845@heistp.net> In-Reply-To: <8E4244B7-C0AE-43D6-980F-C2370E662845@heistp.net> From: Dave Taht Date: Fri, 24 Aug 2018 10:30:22 -0700 Message-ID: To: Pete Heist Cc: bloat , cerowrt-devel@lists.bufferbloat.net Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Subject: Re: [Bloat] beating the drum for BQL X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Aug 2018 17:30:34 -0000 On Fri, Aug 24, 2018 at 10:13 AM Pete Heist wrote: > > > On Aug 23, 2018, at 10:26 AM, Pete Heist wrote: > > On Aug 23, 2018, at 2:49 AM, Dave Taht wrote: > > I had a chance to give a talk at broadcom recently, slides here: > > http://flent-fremont.bufferbloat.net/~d/broadcom_aug9.pdf > > > Thanks for sharing, this is really useful, raising awareness where it mat= ters. Quite a bit of content... :) > > Ubiquiti needs some work getting this into more of their products (EdgeMA= X in particular). A good time to lobby for this might be, well a couple mon= ths ago, as they=E2=80=99re producing alpha builds for their upcoming 2.0 r= elease with kernel 4.9 and new Cavium/Mediatek/Octeon SDKs. I just asked ab= out the status in the EdgeRouter Beta forum, in case it finds the right eye= s before the release: > > https://community.ubnt.com/t5/EdgeRouter-Beta/BQL-support/m-p/2466657 > > > This started a discussion, and no, so far it looks like there=E2=80=99s n= o BQL support in the upcoming 2.0 release. > > For my own benefit, re-reading the original patch series comment (https:/= /lwn.net/Articles/469652/) makes it sound like BQL is useful even without A= QM (original benchmarks were done with straight pfifo_fast). I didn=E2=80= =99t realize this, actually. If anything incorrect about BQL was said in th= is discussion, correct us, please=E2=80=A6 :) yes, bql is very useful even with pfifo fast. without BQL I doubt the internet would be scaling as it is today in the dc, or on the smaller hosts and devices that support it. It's in the mvneta, it's in the ar71xx, with documented results there that I could dig up. (tho: things like tsq are helping and mask the problem on simple tests) The experiment I documented on the slides that kicked off this thread and the other experiment on the systemd bug, easily show the benefit on hosts forwarding packets (be they from local applications, coming from various sources like docker containers, etc), and anyone can show what goes wrong if you disable BQL nowadays, basically restoring linux-3.3 behavior, with a very simple test: For I in /sys/class/net/your_device/queues/tx*/byte_queue_limits/limit_min do echo 10000000 > $I done so long as you run enough kinds of flows that don't engage TSQ. However, in the edgerouter w/offloads case all that part of the stack has been short circuited into the offload engine. I don't know how much buffering is in there on the new firmware, I'd done a few tests on it in the old days, showing it to be around 10ms at gigE but even that memory is kind of vague (the easy test here is slam two ports into one), and for all I know the new firmware is worse, without going back to track this new release. (I do have a few edgerouters but they are all in production) There was also a paper on BQL a few years back that I can dig up.... > Pete > --=20 Dave T=C3=A4ht CEO, TekLibre, LLC http://www.teklibre.com Tel: 1-669-226-2619