From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ob0-x233.google.com (mail-ob0-x233.google.com [IPv6:2607:f8b0:4003:c01::233]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by huchra.bufferbloat.net (Postfix) with ESMTPS id 8EAE121F4F5 for ; Thu, 16 Apr 2015 12:26:03 -0700 (PDT) Received: by oblw8 with SMTP id w8so51671064obl.0 for ; Thu, 16 Apr 2015 12:26:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=JNxijjgX8NzwoRnJmqqMVJeOyT8C3v0gKF8TCg+Bdp4=; b=oIm6mcLJ6v4JSl4yTdTmYGV4IH1vKlXRA56vcvD76nZlhVvKFV06Y2uAJ+IHCb+E3B jtRelRlCFAO+vqKEuWsxu8j30h2W+8S0yFbCQyDUgzRp9+Ne39aa3TcnygXS5bqmmwX/ fpcxav/LOFB178bQJUhsWdsuJoRrWwm8zJFGDMw/JxSJYBMHh01cWO/JDt0PNvCXTdox lpUbZconm1guTpldWIh8lMfO9dj649Q3eKYNZgu3dcI3BRivHhzS4u1UpnWo5RnCH/k5 38enrVNrN+HbIrI4UEPMSoiaiWRQfDOQxXUWTBy4w5sNwKkO7bomdLuiL6qJC+7Lquui jlTg== MIME-Version: 1.0 X-Received: by 10.202.227.130 with SMTP id a124mr21898245oih.59.1429212362347; Thu, 16 Apr 2015 12:26:02 -0700 (PDT) Received: by 10.202.71.139 with HTTP; Thu, 16 Apr 2015 12:26:02 -0700 (PDT) In-Reply-To: References: <5BC1CA30-289D-42B4-95CD-3AE5D7B96F09@gmail.com> Date: Thu, 16 Apr 2015 12:26:02 -0700 Message-ID: From: Dave Taht To: Adrian Popescu Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: cake@lists.bufferbloat.net Subject: Re: [Cake] Cake3 - source code and some questions X-BeenThere: cake@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: Cake - FQ_codel the next generation List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Apr 2015 19:26:31 -0000 On Thu, Apr 16, 2015 at 6:48 AM, Adrian Popescu wrote: > I've discovered there are other problems in the Linux networking stack > which don't seem to be related to fq_codel, qdiscs, AQM and HTB. > > There are latency inducing issues bugs in the Ethernet network drivers > of many network adapters, including e1000e, or in the kernel itself. > Some kernels are better. The newest ones have severe regressions in > this area. There are a multiplicity of problems in doing real-time packet processing while on processors that generally cannot context switch in under 1000 cycles anymore, and on virtual machines, oh, my! One of the biggest fixes for linux networking was the BQL infrastructure wh= ich is in something like 24 drivers now (including the e1000e) - but things lik= e TSO and GRO offloads remain a PITA. There are still many drivers left to fix. One (mvneta) is really bugging me of late.... > I was under the impression there's a problem in codel or fq_codel that > lead to very frequent latency micro-spikes of between 1 and 3 > milliseconds. In my world it is sometimes hard to worry about stuff down in this latency noise level. This is so far less than what these algorithms generally solve in the first place, that I generally have treated it as noise. That said, I remain sad that the linux-rt folk are so underfunded: as they have the tools and expertise to try and get more consistent low latency with full throughput. >It also seemed to produce bigger latency spikes under > moderate load. No amount of tuning and disabling of offloads helped > with this. Regrettably you are not providing enough details. Repeatable tests and actual measurements are always helpful. There is presently what is viewed as a regresson by the Xen folk involving the tcp small queues subsystem, which is being discussed heavily on lkml. What are you referring to specifically? > > Imagine having 2 milliseconds of latency to your ISP and having your > router induce between 3 to 5 milliseconds of latency for every flow. > It's not particularly helpful for low latency paths on high bandwidth > links. Adding more latency in both directions to high latency paths is > even worse. Imagine having your router induce seconds or 10s of seconds latency - the state of affairs on most edge devices today. I DO care about latency and jitter to this level, but it is very hard to isolate and measure. > > These problems are the reason behind starting this thread. I believed > these problems to be related to fq_codel or to the codel algorithm > itself. Not enough detail, what exactly, are you measuring, on what hardware using what tools? > My question about porting these improvements to codel and fq_codel was > strictly about the tighter recovery, better invsqrt and other codel In the tree and in cerowrt for 2 years has been multiple variants of the algorithms under test, individually. Cake rolls up the best of these attempts thus far, and each of those separate models remain in-tree for further testing against all the other variables. In no case have I cared one whit about sub 3ms worth of jitter, I was mostly looking to get faster convergence, better utilization at longer RTTs, better behavior at > 100Mbit, and more filling of the pipe in generally I happen to not agree with jonathon that the better invsqrt (cache) now in = cake accomplishes anything, but plan to test. I do think that the better resumption stuff (which has one part that corrects an error in newton's method going in reverse) helps at > 100mbit, which was a speed we were not able to test effectively at in our prior attempts before we had all these nice test tools. > enhancements mentioned on the wiki page. Well, at the sub 3ms level it is almost always about the device driver, BQL, tcp small queues. and kernel context switch time. There are two feature of BQLs I dislike in that it uses a MIAD (Rather than AIMD) controller, and that it's buffering is additive across hardware multiqueues (and devices have sprouted a lot of those of late, which exhibit birthday problems) I dislike TCP IW10 intensely. I feel the entire GRO/TSO/GSO concepts are basically broken (and that we need to develop better hardware that can deal with packets as packets again). There are quite a large list of things to solve to get latencies lower than 2ms that are *hard*. Perhaps userspace networking is an answer. > The solution to this unstable latency will necessitate migration to > another platform without Linux. I'm aware cake and fq_codel won't fix > this problem on Linux. Well a huge goal here is from the dual BSD/GPL licensing of cake. We hope that someone will show up to port this version of the algorithms to another= OS, and by escaping the monoculture, we will learn more about how to do it more right. There is already a project starting to do a dpdk version Codel is in click, codel in ns2, fq_codel in ns3 - we do not have anyone committed to doing cake in anything else at present. Keep hoping a BSD expert will show up to do a pfsense version. > _______________________________________________ > Cake mailing list > Cake@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/cake --=20 Dave T=C3=A4ht Open Networking needs **Open Source Hardware** https://plus.google.com/u/0/+EricRaymond/posts/JqxCe2pFr67