Re: [Bloat] Kirkwood BQL?

[Alan Jenkins <alan.christopher.jenkins@gmail.com>]

On 29/07/15 18:07, David Lang wrote:
> On Wed, 29 Jul 2015, Alan Jenkins wrote:
>
>> On 29/07/15 12:24, Alan Jenkins wrote:
>>> On 29/07/15 05:32, Rosen Penev wrote:
>>>> Anyone know what the situation is with kirkwood and BQL? I found a
>>>> patch for it but have no idea if there are any issues.
>>>>
>>>> I have such a system but have no idea how to ascertain the efficacy 
>>>> of BQL.
>>>
>>> To the latter:
>>>
>>> BQL works for transmissions that reach the full line rate (e.g. for 
>>> 1000MB ethernet).  It limits the queue that builds in the 
>>> driver/device to the minimum they need.  Then queue mostly builds in 
>>> the generic networking stack, where it can be managed effectively 
>>> e.g. by fq_codel.
>>>
>>> So a simple efficacy test is to run a transmission at full speed, 
>>> and monitor latency (ping) at the same time.  Just make sure the 
>>> device qdisc is set to fq_codel.  fq_codel effectively prioritizes 
>>> ping, so the difference will be very easy to see.
>>>
>>> I don't know if there's any corner cases that want testing as well.
>
> BQL adjusts the number of packets that can be queued based on their 
> size, so you can have far more 64 byte packets queued than you can 
> have 1500 byte packets.
>
> do a ping flood of your network with different packet sizes and look 
> at the queue lengths that are allowed, the queue length should be much 
> higher with small packets.
>
>>> BQL can be disabled at runtime for comparison testing:
>>> http://lists.openwall.net/netdev/2011/12/01/112
>>>
>>> There's a BQL tool to see it working graphically (using readouts 
>>> from the same sysfs directory):
>>> https://github.com/ffainelli/bqlmon
>>>
>>> My Kirkwood setup at home is weak, I basically never reach full link 
>>> speed. So this might be somewhat academic unless you set the link 
>>> speed to 100 or 10 using the ethtool command. (It seems like a good 
>>> idea to test those speeds even if you can do better though).  You 
>>> probably also want to start with offloads (tso, gso, gro) disabled 
>>> using ethtool, because they aggregate packets.
>>>
>>
>> a quick test with a 100M setting, connected to gigabit switch, and 
>> flent tcp_download, shows ping under load increases to about 8ms. 
>> Conclusion: the Debian kirkwood kernel probably isn't doing BQL for 
>> me :).
>
> 8ms of latency under load is doing very well. what are you expecting?
>
> David Lang

Not to sound like half-way across adsl?  I'm describing a LAN test at 100M.

Going against an x86 desktop on gigabit gives 1.2ms.  That's more like 
what I expected.  But there's no bql driver, boo r8169, and 100M gives 
about 4ms.  Killing offloads doesn't seem to help.

On the laptop end I do have bql.  On 100M it goes from about 6ms (with 
bql disabled) to 2ms.  After remembering to disable offloads, bql takes 
it from 1.2ms to 0.7ms.

That final figure sounds good to me.  I kind of expected a more dramatic 
difference with BQL though.

I also found a simple check for bql (without looking at source code).  
The sysfs directories will be present regardless

/sys/class/net/eth0/queues/tx-0/byte_queue_limits/

but if the "limit" file contains "0" even after a test, there's no BQL.