From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wi0-x236.google.com (mail-wi0-x236.google.com [IPv6:2a00:1450:400c:c05::236]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by huchra.bufferbloat.net (Postfix) with ESMTPS id 9D6E921FC27 for ; Wed, 29 Jul 2015 11:51:53 -0700 (PDT) Received: by wibxm9 with SMTP id xm9so39026247wib.1 for ; Wed, 29 Jul 2015 11:51:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=subject:to:references:cc:from:message-id:date:user-agent :mime-version:in-reply-to:content-type:content-transfer-encoding; bh=FzAqTeOx5iGV/OWDnw+fkIIGZOOWxhGxZOED7zbZe3w=; b=Y2FpsNGUOR+UorNJA/tjiT22GYk6+WCyB5z0wjjen8H9FdDcGyrzFzHh6ULYhXA3+Z VZMFoq+1iXR6uHCPwtt3op5cAZqnJHW+9CHMKnlctteAT8mMOO86YV6jMwaVGHLBYyf2 uqfWZYu4+zX+sgwGfBFS97mKCWTHAa1w+gZD/3jgeFV5kq6Cc/Fc11cU9RXM/PYhCNBI fEvrpbu9GrGy5JEq6lCqVpf/RQEegTVrWCr4nGYGYAQZ+432QQTViE7TQ72LN3/gylk7 JUJRzZGKgO10QbkOcyVWE5EDKqcfxtQcUmc+XghsfKMyrZXpsreqNnOdzjkOm157CZFl 4OLw== X-Received: by 10.194.172.130 with SMTP id bc2mr87631365wjc.85.1438195911471; Wed, 29 Jul 2015 11:51:51 -0700 (PDT) Received: from volcano.localdomain (host-89-243-101-162.as13285.net. [89.243.101.162]) by smtp.googlemail.com with ESMTPSA id d7sm25773516wij.0.2015.07.29.11.51.50 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 29 Jul 2015 11:51:50 -0700 (PDT) To: David Lang References: <55B8B7EE.3070401@gmail.com> <55B8BABF.8040708@gmail.com> From: Alan Jenkins Message-ID: <55B920C5.3020201@gmail.com> Date: Wed, 29 Jul 2015 19:51:49 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.1.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Cc: bloat@lists.bufferbloat.net Subject: Re: [Bloat] Kirkwood BQL? X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Jul 2015 18:52:22 -0000 On 29/07/15 18:07, David Lang wrote: > On Wed, 29 Jul 2015, Alan Jenkins wrote: > >> On 29/07/15 12:24, Alan Jenkins wrote: >>> On 29/07/15 05:32, Rosen Penev wrote: >>>> Anyone know what the situation is with kirkwood and BQL? I found a >>>> patch for it but have no idea if there are any issues. >>>> >>>> I have such a system but have no idea how to ascertain the efficacy >>>> of BQL. >>> >>> To the latter: >>> >>> BQL works for transmissions that reach the full line rate (e.g. for >>> 1000MB ethernet). It limits the queue that builds in the >>> driver/device to the minimum they need. Then queue mostly builds in >>> the generic networking stack, where it can be managed effectively >>> e.g. by fq_codel. >>> >>> So a simple efficacy test is to run a transmission at full speed, >>> and monitor latency (ping) at the same time. Just make sure the >>> device qdisc is set to fq_codel. fq_codel effectively prioritizes >>> ping, so the difference will be very easy to see. >>> >>> I don't know if there's any corner cases that want testing as well. > > BQL adjusts the number of packets that can be queued based on their > size, so you can have far more 64 byte packets queued than you can > have 1500 byte packets. > > do a ping flood of your network with different packet sizes and look > at the queue lengths that are allowed, the queue length should be much > higher with small packets. > >>> BQL can be disabled at runtime for comparison testing: >>> http://lists.openwall.net/netdev/2011/12/01/112 >>> >>> There's a BQL tool to see it working graphically (using readouts >>> from the same sysfs directory): >>> https://github.com/ffainelli/bqlmon >>> >>> My Kirkwood setup at home is weak, I basically never reach full link >>> speed. So this might be somewhat academic unless you set the link >>> speed to 100 or 10 using the ethtool command. (It seems like a good >>> idea to test those speeds even if you can do better though). You >>> probably also want to start with offloads (tso, gso, gro) disabled >>> using ethtool, because they aggregate packets. >>> >> >> a quick test with a 100M setting, connected to gigabit switch, and >> flent tcp_download, shows ping under load increases to about 8ms. >> Conclusion: the Debian kirkwood kernel probably isn't doing BQL for >> me :). > > 8ms of latency under load is doing very well. what are you expecting? > > David Lang Not to sound like half-way across adsl? I'm describing a LAN test at 100M. Going against an x86 desktop on gigabit gives 1.2ms. That's more like what I expected. But there's no bql driver, boo r8169, and 100M gives about 4ms. Killing offloads doesn't seem to help. On the laptop end I do have bql. On 100M it goes from about 6ms (with bql disabled) to 2ms. After remembering to disable offloads, bql takes it from 1.2ms to 0.7ms. That final figure sounds good to me. I kind of expected a more dramatic difference with BQL though. I also found a simple check for bql (without looking at source code). The sysfs directories will be present regardless /sys/class/net/eth0/queues/tx-0/byte_queue_limits/ but if the "limit" file contains "0" even after a test, there's no BQL.