From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp145.ord.emailsrvr.com (smtp145.ord.emailsrvr.com [173.203.6.145]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by huchra.bufferbloat.net (Postfix) with ESMTPS id 0EA5921F304 for ; Wed, 11 Jun 2014 18:05:32 -0700 (PDT) Received: from localhost (localhost.localdomain [127.0.0.1]) by smtp11.relay.ord1a.emailsrvr.com (SMTP Server) with ESMTP id 07C38F024A; Wed, 11 Jun 2014 21:05:31 -0400 (EDT) X-Virus-Scanned: OK Received: by smtp11.relay.ord1a.emailsrvr.com (Authenticated sender: dpreed-AT-reed.com) with ESMTPSA id 95E06F0152; Wed, 11 Jun 2014 21:05:30 -0400 (EDT) User-Agent: K-@ Mail for Android X-Priority: 3 In-Reply-To: References: MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----JI1PJHT67AFON769EZ8WXCO2A2TYKC" Content-Transfer-Encoding: 8bit From: "David P. Reed" Date: Wed, 11 Jun 2014 21:05:27 -0400 To: Dave Taht , bloat , "cerowrt-devel@lists.bufferbloat.net" Message-ID: Subject: Re: [Cerowrt-devel] BQL, txqueue lengths and the internet of things X-BeenThere: cerowrt-devel@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: Development issues regarding the cerowrt test router project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Jun 2014 01:05:32 -0000 ------JI1PJHT67AFON769EZ8WXCO2A2TYKC Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset=UTF-8 Maybe you can do a quick blog howto? I'd bet the same could be done for raspberry pi and perhaps my other toy the wandboard which has a gigE adapter and Scsi making it a nice iscsi target or nfs server. De bloating the world... One step at a time. On Jun 11, 2014, Dave Taht wrote: >The bloat problem and solutions are not just limited to fixing >routers, but hosts. > >Nearly every low end board I've seen out there forgos a gigE ethernet >interface in favor of a lower power and cost 100mbit interface. > >No distro I've seen modifies the default pfifo txqueuelen from the >current 1000 packet default down to a more reasonable 100 packet >default in that case. And, while many ethernet devices in this >category are hooked up via usb (and currently hard to add BQL support >to), some are not, and byte queue limit support can be easily added to >those. > >Sadly byte queue limits (BQL) is only implemented on a bunch of top >end ethernet drivers. (about 10, last I looked) > >I needed a break from big problems, so a couple late nights later, I >have a very small patch adding support for BQL to the beaglebone >black: > >http://snapon.lab.bufferbloat.net/~d/0001-Add-BQL-support-to-cpsw-beaglebone-driver.patch > >And the results were quite pleasing at 100mbit. BQL holds things down >to two full size packets in the tx ring and we see an enormous >improvement in bidirectional throughput, jitter, and latency. > >http://snapon.lab.bufferbloat.net/~d/beagle_bql/bql_makes_a_difference.png >http://snapon.lab.bufferbloat.net/~d/beagle_bql/beaglebonewins.png > >The default linux behavior ( pfifo fast, txqueue 1000 ) prior to this >patch looked pretty awful: > >http://snapon.lab.bufferbloat.net/~d/beagle_nobql/pfifo_nobql_tsq3028txqueue1000.svg > >and went to looking like this: > >http://snapon.lab.bufferbloat.net/~d/beagle_bql/pfifo_bql_tsq3028txqueue1000.svg > >And adding the new fq scheduler looked like this: > >http://snapon.lab.bufferbloat.net/~d/beagle_bql/fq_bql_tsq3028.svg > >(fq_codel was similar) > >The fact that we don't achieve full upload throughput on this last >test is probably >due to having a tail dropping switch in the way, and/or some dma >dequeuing >cleanup conflicts between the low level transmit and receive queues on >this device (they share an interrupt AND use napi which seems >puzzling). > >But any day I can get a 4-10x improvement in latency and throughput is >a good day. One IoT device down, thousands to go. It would be nice if >the chipmakers were incorporating bql into boxes destined for the >internet of things. > >-- >Dave Täht >_______________________________________________ >Cerowrt-devel mailing list >Cerowrt-devel@lists.bufferbloat.net >https://lists.bufferbloat.net/listinfo/cerowrt-devel -- Sent from my Android device with K-@ Mail. Please excuse my brevity. ------JI1PJHT67AFON769EZ8WXCO2A2TYKC Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit Maybe you can do a quick blog howto?  I'd bet the same could be done for raspberry pi and perhaps my other toy the wandboard which has a gigE adapter and Scsi making it a nice iscsi target or nfs server.

De bloating the world... One step at a time.

On Jun 11, 2014, Dave Taht <dave.taht@gmail.com> wrote:
The bloat problem and solutions are not just limited to fixing
routers, but hosts.

Nearly every low end board I've seen out there forgos a gigE ethernet
interface in favor of a lower power and cost 100mbit interface.

No distro I've seen modifies the default pfifo txqueuelen from the
current 1000 packet default down to a more reasonable 100 packet
default in that case. And, while many ethernet devices in this
category are hooked up via usb (and currently hard to add BQL support
to), some are not, and byte queue limit support can be easily added to
those.

Sadly byte queue limits (BQL) is only implemented on a bunch of top
end ethernet drivers. (about 10, last I looked)

I needed a break from big problems, so a couple late nights later, I
have a very small patch adding support for BQL to the beaglebone
black:

http://snapon.lab.bufferbloat.net/~d/0001-Add-BQL-support-to-cpsw-beaglebone-driver.patch

And the results were quite pleasing at 100mbit. BQL holds things down
to two full size packets in the tx ring and we see an enormous
improvement in bidirectional throughput, jitter, and latency.

http://snapon.lab.bufferbloat.net/~d/beagle_bql/bql_makes_a_difference.png
http://snapon.lab.bufferbloat.net/~d/beagle_bql/beaglebonewins.png

The default linux behavior ( pfifo fast, txqueue 1000 ) prior to this
patch looked pretty awful:

http://snapon.lab.bufferbloat.net/~d/beagle_nobql/pfifo_nobql_tsq3028txqueue1000.svg

and went to looking like this:

http://snapon.lab.bufferbloat.net/~d/beagle_bql/pfifo_bql_tsq3028txqueue1000.svg

And adding the new fq scheduler looked like this:

http://snapon.lab.bufferbloat.net/~d/beagle_bql/fq_bql_tsq3028.svg

(fq_codel was similar)

The fact that we don't achieve full upload throughput on this last
test is probably
due to having a tail dropping switch in the way, and/or some dma dequeuing
cleanup conflicts between the low level transmit and receive queues on
this device (they share an interrupt AND use napi which seems
puzzling).

But any day I can get a 4-10x improvement in latency and throughput is
a good day. One IoT device down, thousands to go. It would be nice if
the chipmakers were incorporating bql into boxes destined for the
internet of things.

-- Sent from my Android device with K-@ Mail. Please excuse my brevity. ------JI1PJHT67AFON769EZ8WXCO2A2TYKC--