From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wg0-f47.google.com (mail-wg0-f47.google.com [74.125.82.47]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client did not present a certificate) by huchra.bufferbloat.net (Postfix) with ESMTPS id 43FD3201B71 for ; Sun, 2 Sep 2012 11:17:18 -0700 (PDT) Received: by wgbfa7 with SMTP id fa7so2622089wgb.28 for ; Sun, 02 Sep 2012 11:17:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=Y7+ydDhIVQGeYUFcOXUTF3G1AbsP2OjJIFuXKzrSlFI=; b=MSC1YdQ0pR0CjRW55/awJWluxH4g8DQ+0PE4TS5iyRiDYPJhHTZdDLA5a6MYZDCztK izgp2758zQ3p2X0AMZrdAO/nMpgIbzqR84AH2SDjboEa1y1O4jQQUQj6UGAx5MUV9ZSE s6UX3Ep282z0B7kZChnGC7z/0Q1yahbdSPQ/TbPOAxy4IvZNlD6t5Zjnekov/d9EWkuR JUTn3HYYw8MhE35viNQt6ULi+daJUjZRLEFDoPilAZaSjhrm27iTnFRaYbKOWju3DjFR qwHeFaPuF/cefG3Jm5+rHqOJVTSYvSqwbzH+OpV606MAyHusdtakYmlDfCOg6XIzS07l DETw== MIME-Version: 1.0 Received: by 10.180.77.34 with SMTP id p2mr17948515wiw.0.1346609836483; Sun, 02 Sep 2012 11:17:16 -0700 (PDT) Received: by 10.223.159.134 with HTTP; Sun, 2 Sep 2012 11:17:16 -0700 (PDT) In-Reply-To: References: <1346396137.2586.301.camel@edumazet-glaptop> <5040DDE9.7030507@hp.com> <1346430207.7996.11.camel@edumazet-glaptop> <1346504012.7996.68.camel@edumazet-glaptop> Date: Sun, 2 Sep 2012 11:17:16 -0700 Message-ID: From: Dave Taht To: Eric Dumazet Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: codel@lists.bufferbloat.net Subject: Re: [Codel] fq_codel : interval servo X-BeenThere: codel@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: CoDel AQM discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 02 Sep 2012 18:17:18 -0000 On Sun, Sep 2, 2012 at 11:08 AM, Dave Taht wrote: In reviewing this mail I realized I used three different names for tcp_limit_output_bytes, corrected below... > Codel will push stuff down to, but not below, 5ms of latency (or > target). In fq_codel you will typically end up with 1 packet outstanding = in > each active queue under heavy load. At 10Mbit it's pretty easy to > have it strain mightily and fail to get to 5ms, particularly on torrent-l= ike > workloads. > > The "right" amount of host latency to aim for is ... 0, or as close to it= as > you can get. Fiddling with codel target and interval on the host to > get less host latency is well and good, but you can't get to 0 that way..= . > > The best queue on a host is no extra queue. > > I spent some time evaluating linux fq_codel vs the ns2 nfq_codel version = I > just got working. In 150 bidirectional competing streams, at 100Mbit, > it retained about 30% less packets in queue (110 vs 140). Next up > on my list is longer RTTs and wifi, but all else was pretty equivalent. > > The effects of fiddling with /proc/sys/net/ipv4/tcp_limit_output_bytes > was even more remarkable. At 6000, I would get down to > a nice steady 71-81 packets in queue on that 150 stream workload. > > So, I started thinking through and playing with how TSQ works: > > At one hop 100Mbit, with a BQL of 3000 and a tcp_limit_output_bytes of 60= 00, > all offloads off, nfq_codel on both ends, I get single stream throughoutp= ut > of 92.85Mbit. Backlog in qdisc is, 0. > > 2 netperf streams, bidirectional: 91.47 each, darn close to theoretical, = less > than one packet in the backlog. > > 4 streams backlogs a little over 3. (and sums to 91.94 in each direction) > > 8, backlog of 8. (optimal throughput) > > Repeating the 8 stream test with tcp_limit_output_bytes of 1500, I get > packets outstanding of around 3, and optimal throughput. (1 stream test: > 42Mbit throughput (obviously starved), 150 streams: 82...) > > 8 streams, limit set to 127k, I get 50 packets outstanding in the queue, > and the same throughput. (150 streams, ~100) > > So I might argue that a more "right" number for tcp_limit_output_bytes is > not 128k per TCP socket, but (BQL_limit*2/active_sockets), in conjunction > with fq_codel. I realize that that raises interesting questions as to whe= n > to use TSO/GSO, and how to schedule tcp packet releases, and pushes > the window reduction issue all the way up into the tcp stack rather > than responding to indications from the qdisc... but it does > get you closer to a 0 backlog in qdisc. > > And *usually* the bottleneck link is not on the host but on something > inbetween, and that's where your signalling comes from, anyway. > > > -- > Dave T=E4ht > http://www.bufferbloat.net/projects/cerowrt/wiki - "3.3.8-17 is out > with fq_codel!" --=20 Dave T=E4ht http://www.bufferbloat.net/projects/cerowrt/wiki - "3.3.8-17 is out with fq_codel!"