From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ie0-f171.google.com (mail-ie0-f171.google.com [209.85.223.171]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority" (verified OK)) by huchra.bufferbloat.net (Postfix) with ESMTPS id C56C121F166 for ; Wed, 21 Nov 2012 23:22:31 -0800 (PST) Received: by mail-ie0-f171.google.com with SMTP id 17so7672878iea.16 for ; Wed, 21 Nov 2012 23:22:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type :content-transfer-encoding; bh=lBkdva1ISK7yiiKsKBAsFspSVUnd5DvMor9qr01nAeU=; b=Ra+3CgzOMKJGUfQKdomDsblb6d9AosXPADwULYJRnnRq4kneVdg9rgrOV7ns/RhiGP ie7rgoHgNNL1vSg9oN9o9oF/P9X8b4NxIYZyGhEfVMW/auK43FhFzoIhQJf/YiPBkxWq pTguNFQV1XVyxqcuLcfAOPXEZxlhS5h8P3w8EasaaIMSvxbkFE1SATMEABus3J0oBDw0 HGGoo6Iav1VdQwyHouvt/T1ofqRiWhGprtBQvRrm/abh7q35Z3v2YO16dkB+jeXu2/hc fsXTZ2/odLINvrk6+WU0z6x5ABMAFeuce+dO5c+nLvDtvMSTVARjqg1QI2C3bbQmVQh8 CeVw== MIME-Version: 1.0 Received: by 10.43.92.72 with SMTP id bp8mr19813013icc.49.1353568950975; Wed, 21 Nov 2012 23:22:30 -0800 (PST) Received: by 10.64.135.39 with HTTP; Wed, 21 Nov 2012 23:22:30 -0800 (PST) Date: Thu, 22 Nov 2012 08:22:30 +0100 Message-ID: From: Dave Taht To: codel@lists.bufferbloat.net, David Woodhouse Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Subject: [Codel] GRO (generic receive offload) is not helpful with codel and fq_codel X-BeenThere: codel@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: CoDel AQM discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Nov 2012 07:22:32 -0000 I am fiddling with one of david woodhouse's machines (which is 16MBit down, .75Mbit up) Pouring packets through it, I noticed significant jitter while using HTB to rate limit, and saw maxpacket consistently hitting 14546, rather than staying at a MTU. Normally, when I see this, this is due to TSO/GSO offloads which I've turned off on this machine's egress network card. But as this is happening when I'm forwarding packets I thought that perhaps it was GRO. But I'd turned that off too, on this network card and hard limited BQL to my usual 3k, and I still hit it.= ... tc -s qdisc show dev eth1 ... snip, snip... qdisc fq_codel 120: parent 1:12 limit 10240p flows 16000 quantum 500 target 20.0ms interval 100.0ms Sent 2213780 bytes 4843 pkt (dropped 48, overlimits 0 requeues 0) backlog 12160b 8p requeues 0 maxpacket 14546 drop_overlimit 0 new_flow_count 1237 ecn_mark 0 ******* new_flows_len 1 old_flows_len 2 Then! (after sleeping on it) I realized that the source of the overlarge packets was coming from the *main* ethernet interface, which were then getting pumped through the egress interface. So turning off GRO on THAT (eth0) interface, got maxpacket down to the sane size of 1514. And latency and jitter - and for that matter, inter-flow fairness - markedly improved. I went from induced latency under load (pre-fq_codel) of 350 ms on this link, to 0-90ms with GRO on, to 0-30ms with it off. I will try and get around to doing a cdf plot of the overall differences today, because the median looked closer to 70 with GRO on and closer to 20 with GRO off. This was 3.6.6-1.fc16.i686 with a realtek 8169. I don't remember if this is BQLed or not (doesn't matter in this test case, as I am using HTB) [ 16.589197] r8169 eth1: RTL8169sc/8110sc So, those of you that are testing codel and fq_codel, etc, might want to see what maxpacket is hitting on your setup. That seems explain some fq_codel jitter and codel performance issues in several respects in the testing people are doing at low bandwidths. You can turn off offloads via ethtool -K eth1 tso off off gro off ethtool -k eth1 Offload parameters for eth1: rx-checksumming: on tx-checksumming: off scatter-gather: off tcp-segmentation-offload: off udp-fragmentation-offload: off generic-segmentation-offload: off generic-receive-offload: off large-receive-offload: off rx-vlan-offload: on tx-vlan-offload: on ntuple-filters: off receive-hashing: off Lastly I note that given the very slow uplink I was also fiddling with the codel target value. What effect that actually has is unknown, more testing today - but I think perhaps I was debugging the wrong problem, and it was GRO on one of the interfaces causing the oddities. My thanks to David Woodhouse for letting me hack on his home system... --=20 Dave T=E4ht Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.= html