From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wi0-f169.google.com (mail-wi0-f169.google.com [209.85.212.169]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority" (verified OK)) by huchra.bufferbloat.net (Postfix) with ESMTPS id 927BB200B1E for ; Fri, 6 Apr 2012 14:49:40 -0700 (PDT) Received: by wibhm17 with SMTP id hm17so836116wib.4 for ; Fri, 06 Apr 2012 14:49:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=hJxb01bSN5hcpwfBYn0GPdfxaG3cIq0ChS7hJyqERnU=; b=gQx6udr65iFmyqcFxZTvKfl3kf+Hz1VxRdPM60UJQTjz7HUwRC2ZxJUzHCyxGiRYVO Iof8CW+U2s4bY+lfdQ1AMvMXWr8C29lxIU0HMlpWGOat8frZvdstx4s0rkzw/VHMrwD2 qImYhdVqzAusVU9KqB7FC8Wbk7QJBwAWThFyy53g4RxBbhZWCvLNaiV2n7h/Fqb3QHPP O7+owyW92GPYg4tOHJRxDAz5KDr0m7VrSyV1U+iuM89oeJKAPM74stxRjDV6b90FZiP0 uP4g8UD3zYE96XE3f9CpKavyTRM3Mi8M0ku+ZaYpjA5WJe5WaDIr64DKhrxRGrYGrlSJ eE1g== MIME-Version: 1.0 Received: by 10.180.95.34 with SMTP id dh2mr14257709wib.15.1333748978640; Fri, 06 Apr 2012 14:49:38 -0700 (PDT) Received: by 10.223.127.194 with HTTP; Fri, 6 Apr 2012 14:49:38 -0700 (PDT) In-Reply-To: <20120406213725.GA12641@uio.no> References: <20120406213725.GA12641@uio.no> Date: Fri, 6 Apr 2012 14:49:38 -0700 Message-ID: From: Dave Taht To: "Steinar H. Gunderson" Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: bloat@lists.bufferbloat.net Subject: Re: [Bloat] Best practices for paced TCP on Linux? X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Apr 2012 21:49:41 -0000 On Fri, Apr 6, 2012 at 2:37 PM, Steinar H. Gunderson wrote: > Hi, > > This is only related to bloat, so bear with me if it's not 100% on-topic; > I guess the list is about the best place on the Internet to get a reasonb= le > answer for this anyway :-) This is actually both very relevant and very hard to simulate. Having someone willing to try this in the real world would be excellent. > Long story short, I have a Linux box (running 3.2.0 or so) with a 10Gbit/= sec > interface, streaming a large amount of video streams to external users, > at 1Mbit/sec, 3Mbit/sec or 5Mbit/sec (different values). Unfortunately, e= ven > though there is no congestion in _our_ network (we have 190 Gbit/sec free= !), > some users are complaining that they can't keep the stream up. Interesting. While there can be many causes I do tend to think that 10GigE is putting incredible pressure on downstream devices that is very hard to observe, unless you are downstream... > > My guess is that this is because at 10Gbit/sec, we are crazy bursty, and > somewhere along the line, there will be devices doing down conversion wit= hout > enough buffers (for instance, I've seen drop behavior on Cisco 2960-S in = a > very real ISP network on 10->1 Gbit/sec down conversion, and I doubt it's= the > worst offender here). > > Is there anything I can do about this on my end? I looked around for pace= d > TCP implementations, but couldn't find anything current. Can I somehow sh= ape > each TCP stream to 10Mbit/sec or so each with a combination of SFQ and TB= F? > (SFQRED?) It would be best to get some packet captures from user's streams that are complaining of the issue. Most 10Gige network cards have the ability to have multiple hardware queues that will do some FQ for you. SFQ or QFQ will FQ the output streams. SFQRED or QFQ + something will be able to do some level of queue management but not 'shaping', or 'packet pac= ing' However in your environment you will need the beefed up SFQ that is in 3.3. and BQL. If you are not saturating that 10GigE card, you can turn off TSO/G= SO as well. You can reduce your tcp sent windows. There's a horde of other things you can try, but it would be best to start of with a few measurements, both on your side and at a customer's. > I'm not very well versed in tc, so anything practical would be very much I have a debloat script (with about 5 out of tree versions) that tries it's= best to do the right thing on home gear and takes a bit of the arcaneness out of the mix. I have not got around to trying out a couple algos > appreciated. Bonus points if we won't have to patch the kernel. 3.3.1 is out and in my environment is stable. YMMV. > /* Steinar */ > -- > Homepage: http://www.sesse.net/ > _______________________________________________ > Bloat mailing list > Bloat@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/bloat --=20 Dave T=E4ht SKYPE: davetaht US Tel: 1-239-829-5608 http://www.bufferbloat.net