From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pd0-x230.google.com (mail-pd0-x230.google.com [IPv6:2607:f8b0:400e:c02::230]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority" (verified OK)) by huchra.bufferbloat.net (Postfix) with ESMTPS id D8C0221F185; Tue, 9 Jul 2013 05:56:35 -0700 (PDT) Received: by mail-pd0-f176.google.com with SMTP id t12so5194590pdi.21 for ; Tue, 09 Jul 2013 05:56:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:subject:from:to:cc:date:in-reply-to:references :content-type:x-mailer:content-transfer-encoding:mime-version; bh=USwL8jVV8gsXA5yneZKknxRJwvDt3De7jLwPqlcHNxI=; b=DAyJkAW6UyL4imhlelOUAZ1oOOz1BpeJKBlbbdlSY00EOMNlTmEUm1pJQait83qkkp FwtgvQCXAljQG++34R4vyzoKxYWSp2EXh24vBpg4GF3oPbSiwLm8hFHvbS0MMD6O2biZ F1EelPOBNujf2/7w9kpMSUWtveRWLkdJUc0rZ8zqtGAlf+HlLrhuc7s7nWIwZ6pgUGhB TIGPpJWU9RFwIQ/yQstZxvf6VtVHlzOS60Z8bs/uiDVSYV6YWOytdqNKh8RB0xbK1tLO cg+I7omq6Kj6SNuTFCs0s9eI4gem7+6pT++o+CRnqO5keMj6tj09mQJb/8KVrUmUD4ag 1/uw== X-Received: by 10.68.203.161 with SMTP id kr1mr26288823pbc.192.1373374595267; Tue, 09 Jul 2013 05:56:35 -0700 (PDT) Received: from [172.19.248.202] ([172.19.248.202]) by mx.google.com with ESMTPSA id s5sm28142660pbo.38.2013.07.09.05.56.34 for (version=SSLv3 cipher=RC4-SHA bits=128/128); Tue, 09 Jul 2013 05:56:34 -0700 (PDT) Message-ID: <1373374593.4979.142.camel@edumazet-glaptop> From: Eric Dumazet To: Toke =?ISO-8859-1?Q?H=F8iland-J=F8rgensen?= Date: Tue, 09 Jul 2013 05:56:33 -0700 In-Reply-To: <87obacw4gv.fsf@toke.dk> References: <1373223178.486913695@apps.rackspace.com> <871u79x9kb.fsf@toke.dk> <87obacw4gv.fsf@toke.dk> Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.2.3-0ubuntu6 Content-Transfer-Encoding: 8bit Mime-Version: 1.0 Cc: codel@lists.bufferbloat.net, cerowrt-devel@lists.bufferbloat.net Subject: Re: [Cerowrt-devel] [Codel] happy 4th! X-BeenThere: cerowrt-devel@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: Development issues regarding the cerowrt test router project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Jul 2013 12:56:36 -0000 On Tue, 2013-07-09 at 09:57 +0200, Toke Høiland-Jørgensen wrote: > Mikael Abrahamsson writes: > > > For me, it shows that FQ_CODEL indeed affects TCP performance > > negatively for long links, however it looks like the impact is only > > about 20-30%. > > As far as I can tell, fq_codel's throughput is about 10% lower on > 100mbit in one direction, while being higher in the other. For 10mbit > fq_codel shows higher throughput throughout? What do you mean ? This makes little sense to me. > > > What's stranger is that latency only goes up to around 230ms from its > > 200ms "floor" with FIFO, I had expected a bigger increase in buffering > > with FIFO. Have you done any TCP tuning? > > Not apart from what's in mainline (3.9.9 kernel). The latency-inducing > box is after the bottleneck, though, so perhaps it has something to do > with that? Some interaction between netem and the ethernet link? I did not received a copy of your setup, so its hard to tell. But using netem correctly is tricky. My current testbed uses the following script, meant to exercise tcp flows with random RTT between 49.9 and 50.1 ms, to check how TCP stack reacts to reorders (The answer is : pretty badly.) Note that using this setup forced me to send two netem patches, currently in net-next, or else netem used too many cpu cycles on its own. http://git.kernel.org/cgit/linux/kernel/git/davem/net-next.git/commit/?id=aec0a40a6f78843c0ce73f7398230ee5184f896d Followed by a fix : http://git.kernel.org/cgit/linux/kernel/git/davem/net-next.git/commit/?id=36b7bfe09b6deb71bf387852465245783c9a6208 The script : # netem based setup, installed at receiver side only ETH=eth4 IFB=ifb0 EST="est 1sec 4sec" # Optional rate estimator modprobe ifb ip link set dev $IFB up tc qdisc add dev $ETH ingress 2>/dev/null tc filter add dev $ETH parent ffff: \ protocol ip u32 match u32 0 0 flowid 1:1 action mirred egress \ redirect dev $IFB ethtool -K $ETH gro off lro off tc qdisc del dev $IFB root 2>/dev/null # Use netem at ingress to delay packets by 25 ms +/- 100us (to get reorders) tc qdisc add dev $IFB root $EST netem limit 100000 delay 25ms 100us # loss 0.1 tc qdisc del dev $ETH root 2>/dev/null # Use netem at egress to delay packets by 25 ms (no reorders) tc qd add dev $ETH root $EST netem delay 25ms limit 100000 And the results for a single tcp flow : lpq84:~# ./netperf -H 10.7.7.83 -l 10 MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.7.7.83 () port 0 AF_INET Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec 87380 16384 16384 10.20 37.60 lpq84:~# ./netperf -H 10.7.7.83 -l 10 MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.7.7.83 () port 0 AF_INET Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec 87380 16384 16384 10.06 116.94 See rates at the receiver side (check if packets were dropped because of too low qdisc limits, and rates) lpq83:~# tc -s -d qd qdisc netem 800e: dev eth4 root refcnt 257 limit 100000 delay 25.0ms Sent 10791616 bytes 115916 pkt (dropped 0, overlimits 0 requeues 0) rate 7701Kbit 10318pps backlog 47470b 509p requeues 0 qdisc ingress ffff: dev eth4 parent ffff:fff1 ---------------- Sent 8867475174 bytes 5914081 pkt (dropped 0, overlimits 0 requeues 0) backlog 0b 0p requeues 0 qdisc netem 800d: dev ifb0 root refcnt 2 limit 100000 delay 25.0ms 99us Sent 176209244 bytes 116430 pkt (dropped 0, overlimits 0 requeues 0) rate 123481Kbit 10198pps backlog 0b 0p requeues 0