From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-we0-x230.google.com (mail-we0-x230.google.com [IPv6:2a00:1450:400c:c03::230]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by huchra.bufferbloat.net (Postfix) with ESMTPS id 48DE021F2FC; Wed, 11 Jun 2014 15:49:46 -0700 (PDT) Received: by mail-we0-f176.google.com with SMTP id u56so417833wes.7 for ; Wed, 11 Jun 2014 15:49:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type :content-transfer-encoding; bh=7GJrp7c1dJ7kXYRTzX8BVKAi0GuSbefDos6A8infVgk=; b=rMQMlxsGR+rI2OArEYEdn1idb933NyfZz7BcPIt94pKnyW9OEefAMVDtLSmU1FY3Lm BU2Y+QlSi1o1EgWA+P91Bl2z4x9QYn5KX0XxrtUG88bctUWvXcwlVyIIDS8JE4/yFsks XTGyDkxdInQZ33a2dd2Xwgu60pA0ALBxnmE0HvbpiZjqYlog7zMbOYTcE1c1OK19NxGn ojTEHztfib8zqUbblot081y0bgEFHx8l9cFWHj+bnYwre/DfZ0UJLWsXByf0lNyO3JiO R66/tvDhBY1U1VmzIe0qIM2CRWMjiaZfiFpOJd/ED5IlIXzHj02ocWD1wagM70hBvzGv cK7A== MIME-Version: 1.0 X-Received: by 10.180.149.175 with SMTP id ub15mr815582wib.53.1402526983698; Wed, 11 Jun 2014 15:49:43 -0700 (PDT) Received: by 10.216.207.82 with HTTP; Wed, 11 Jun 2014 15:49:43 -0700 (PDT) Date: Wed, 11 Jun 2014 15:49:43 -0700 Message-ID: From: Dave Taht To: bloat , "cerowrt-devel@lists.bufferbloat.net" Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Subject: [Cerowrt-devel] BQL, txqueue lengths and the internet of things X-BeenThere: cerowrt-devel@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: Development issues regarding the cerowrt test router project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Jun 2014 22:49:46 -0000 The bloat problem and solutions are not just limited to fixing routers, but hosts. Nearly every low end board I've seen out there forgos a gigE ethernet interface in favor of a lower power and cost 100mbit interface. No distro I've seen modifies the default pfifo txqueuelen from the current 1000 packet default down to a more reasonable 100 packet default in that case. And, while many ethernet devices in this category are hooked up via usb (and currently hard to add BQL support to), some are not, and byte queue limit support can be easily added to those. Sadly byte queue limits (BQL) is only implemented on a bunch of top end ethernet drivers. (about 10, last I looked) I needed a break from big problems, so a couple late nights later, I have a very small patch adding support for BQL to the beaglebone black: http://snapon.lab.bufferbloat.net/~d/0001-Add-BQL-support-to-cpsw-beaglebon= e-driver.patch And the results were quite pleasing at 100mbit. BQL holds things down to two full size packets in the tx ring and we see an enormous improvement in bidirectional throughput, jitter, and latency. http://snapon.lab.bufferbloat.net/~d/beagle_bql/bql_makes_a_difference.png http://snapon.lab.bufferbloat.net/~d/beagle_bql/beaglebonewins.png The default linux behavior ( pfifo fast, txqueue 1000 ) prior to this patch looked pretty awful: http://snapon.lab.bufferbloat.net/~d/beagle_nobql/pfifo_nobql_tsq3028txqueu= e1000.svg and went to looking like this: http://snapon.lab.bufferbloat.net/~d/beagle_bql/pfifo_bql_tsq3028txqueue100= 0.svg And adding the new fq scheduler looked like this: http://snapon.lab.bufferbloat.net/~d/beagle_bql/fq_bql_tsq3028.svg (fq_codel was similar) The fact that we don't achieve full upload throughput on this last test is probably due to having a tail dropping switch in the way, and/or some dma dequeuing cleanup conflicts between the low level transmit and receive queues on this device (they share an interrupt AND use napi which seems puzzling). But any day I can get a 4-10x improvement in latency and throughput is a good day. One IoT device down, thousands to go. It would be nice if the chipmakers were incorporating bql into boxes destined for the internet of things. --=20 Dave T=C3=A4ht