From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ia0-f171.google.com (mail-ia0-f171.google.com [209.85.210.171]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority" (verified OK)) by huchra.bufferbloat.net (Postfix) with ESMTPS id B7FDD201B88; Fri, 23 Nov 2012 00:57:35 -0800 (PST) Received: by mail-ia0-f171.google.com with SMTP id b35so4817895iac.16 for ; Fri, 23 Nov 2012 00:57:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=0MO9xJESUATHsoBdU6F+PQ3nc4CneBx8nUcnhwUF3bk=; b=EH7e75S+tagq+LMEsVpwGvouVkpMq+HUA/zCEShjwiWz2FKJJhsqEgZGRk0vb8/hdQ 8H5bsB2m4ZVWnAhbwds/Gi8+ZN1CRfdoqWjcAoTcQcahyPeSw/A0KWuVZMXqTNnyXw2f 53NfyWHnZCpPO9SlEbC4wz3553taMd0Le4f+Oz5+fIe2E28pOezrzsPzKEkwKdlovwzz 0Ji88LTS0oz4RRNCYwvk9BItcJFgiapQVNQF78vfFOukVlEgsq7dDK+0iZ884ru9v0O9 pRswkzMHlPOf2IEPIf4hZl8kY0alWl7rH4GRX9EAfHy4jBiLw4e/4uz3VF5FhMB/HuMq c7nQ== MIME-Version: 1.0 Received: by 10.50.150.144 with SMTP id ui16mr2915060igb.68.1353661054640; Fri, 23 Nov 2012 00:57:34 -0800 (PST) Received: by 10.64.135.39 with HTTP; Fri, 23 Nov 2012 00:57:34 -0800 (PST) In-Reply-To: References: Date: Fri, 23 Nov 2012 09:57:34 +0100 Message-ID: From: Dave Taht To: paulmck@linux.vnet.ibm.com, bloat , cerowrt-devel@lists.bufferbloat.net, codel@lists.bufferbloat.net Content-Type: text/plain; charset=ISO-8859-1 Cc: Eric Raymond , Paolo Valente , =?ISO-8859-1?Q?Toke_H=F8iland=2DJ=F8rgensen?= , John Crispin Subject: Re: [Codel] FQ_Codel lwn draft article review X-BeenThere: codel@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: CoDel AQM discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Nov 2012 08:57:36 -0000 David Woodhouse and I fiddled a lot with adsl and openwrt and a variety of drivers and network layers in a typical bonded adsl stack yesterday. The complexity of it all makes my head hurt. I'm happy that a newly BQL'd ethernet driver (for the geos and qemu) emerged from it, which he submitted to netdev... I made a recording of us last night discussing the layers, which I will produce and distribute later... Anyway, along the way, we fiddled a lot with trying to analyze where the 350ms or so of added latency was coming from in the traverse geo's adsl implementation and overlying stack.... Plots: http://david.woodhou.se/dwmw2-netperf-plots.tar.gz Note: 1: The netperf sample rate on the rrul test needs to be higher than 100ms in order to get a decent result at sub 10Mbit speeds. Note 2: The two nicest graphs here are nofq.svg vs fq.svg, which were taken on a gigE link from a Mac running Linux to another gigE link. (in other words, NOT on the friggin adsl link) (firefox can display svg, I don't know what else) I find the T+10 delay before stream start in the fq.svg graph suspicious and think the "throw out the outlier" code in the netperf-wrapper code is at fault. Prior to that, codel is merely buffering up things madly, which can also be seen in the pfifo_fast behavior, with 1000pkts it's default. (Arguably, the default queue length in codel can be reduced from 10k packets to something more reasonable at GigE speeds) (the indicator that it's the graph, not the reality, is that the fq.svg pings and udp start at T+5 and grow minimally, as is usual with fq_codel.) As for the *.ps graphs, well, they would take david's network topology to explain, and were conducted over a variety of circumstances, including wifi, with more variables in play than I care to think about. We didn't really get anywhere on digging deeper. As we got to purer tests - with a minimal number of boxes, running pure ethernet, switched over a couple of switches, even in the simplest two box case, my HTB based "ceroshaper" implementation had multiple problems in cutting median latencies below 100ms, on this very slow ADSL link. David suspects problems on the path along the carrier backbone as a potential issue, and the only way to measure that is with two one way trip time measurements (rather than rtt), time synced via ntp... I keep hoping to find a rtp test, but I'm open to just about any option at this point. anyone? We also found a probable bug in mtr in that multiple mtrs on the same box don't co-exist. Moving back to more scientific clarity and simpler tests... The two graphs, taken a few weeks back, on pages 5 and 6 of this: http://www.teklibre.com/~d/bloat/Not_every_packet_is_sacred-Battling_Bufferbloat_on_wifi.pdf appear to show the advantage of fq_codel fq + codel + head drop over tail drop during the slow start period on a 10Mbit link - (see how squiggly slow start is on pfifo fast?) as well as the marvelous interstream latency that can be achieved with BQL=3000 (on a 10 mbit link.) Even that latency can be halved by reducing BQL to 1500, which is just fine on a 10mbit. Below those rates I'd like to be rid of BQL entirely, and just have a single packet outstanding... in everything from adsl to cable... That said, I'd welcome other explanations of the squiggly slowstart pfifo_fast behavior before I put that explanation on the slide.... ECN was in play here, too. I can redo this test easily, it's basically running a netperf TCP_RR for 70 seconds, and starting up a TCP_MAERTS and TCP_STREAM for 60 seconds a T+5, after hammering down on BQL's limit and the link speeds on two sides of a directly connected laptop connection. ethtool -s eth0 advertise 0x002 # 10 Mbit