From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wi0-f175.google.com (mail-wi0-f175.google.com [209.85.212.175]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority" (verified OK)) by huchra.bufferbloat.net (Postfix) with ESMTPS id D0E8C200B33 for ; Sat, 7 Apr 2012 12:38:06 -0700 (PDT) Received: by wibhn6 with SMTP id hn6so1213474wib.10 for ; Sat, 07 Apr 2012 12:38:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=1SyuZQNuQXFr27zAEuvG1RsCzhJvx1zUxCvdjCOpk2Q=; b=Riwe1EUUJAELrBcqZY6d9WohYrYqLGkM5k+DJ07f6fWUCr4aAfgsCxLAAysqU/F/p9 FeDIp/IncxcHuZ/7lE879nIIgsymOwDY2a9qjBdjLaGPHfbQCPo7Q2pRF9N7+D98jreU ryz/n/iRlc15dCDEO0XmGSQB3jUYIvoVvDUAZHSBqBB1J2KHnKAmYsnZCZbRmoFTvuxQ 0IxrAk7L63wt2syUjfJxSqsfoUx8bsS5jOV6wZTxtMRX9Rwf1JIddat4tNQenfeUm0Lt 1EaW8t+9rYvfMFzLA925No+mJzPyCKJZGHJc7seFGLDM0VJBBwaL+ku/Y6G1sfPsUNCE NkPg== MIME-Version: 1.0 Received: by 10.180.95.74 with SMTP id di10mr5776102wib.1.1333827484530; Sat, 07 Apr 2012 12:38:04 -0700 (PDT) Received: by 10.223.127.194 with HTTP; Sat, 7 Apr 2012 12:38:04 -0700 (PDT) In-Reply-To: <20120407190134.GF21452@uio.no> References: <20120406213725.GA12641@uio.no> <20120406222138.GB12641@uio.no> <1333811327.30705.4.camel@edumazet-laptop> <20120407153548.GC21452@uio.no> <20120407181034.GD21452@uio.no> <20120407185456.GE21452@uio.no> <20120407190134.GF21452@uio.no> Date: Sat, 7 Apr 2012 12:38:04 -0700 Message-ID: From: Dave Taht To: "Steinar H. Gunderson" Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: bloat@lists.bufferbloat.net Subject: Re: [Bloat] Best practices for paced TCP on Linux? X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 07 Apr 2012 19:38:07 -0000 On Sat, Apr 7, 2012 at 12:01 PM, Steinar H. Gunderson wrote: > On Sat, Apr 07, 2012 at 08:54:56PM +0200, Steinar H. Gunderson wrote: >> I did these on one of the irrors (well, I did 500000 instead of 256000). >> You can try >> >> =A0 http://pannekake.samfundet.no:3013/ (SD) >> =A0 http://pannekake.samfundet.no:3015/ (HD) >> >> I didn't restart VLC; I hope I don't have to. =3D) > > I got reports from people in Norway that this instantly stopped the probl= ems > on the HD stream, so incredibly enough, it may have worked. > > I don't understand these mechanisms. Why would a smaller send window help= ? > Less burstiness? Awesome. I still think it's way too big, but there's some divisor in here (1/4?) that I don't remember. As for an explanation... Welcome to bufferbloat, the global plague that is sweeping the world! Up until you hit the available bandwidth on a path, life is golden, and response time to lost packets approximately equals the overall latency in the path, say, 10ms for around town. Your video player has at least a few 100ms worth of it's own buffering, so it doesn't even notice anything but a truly massive outage. TCP just recovers, transparently, underneat. But: You pass that operating point, all the buffers in the path fill, your delays go way up, then you finally lose a packet (see rfc970) and tcp cannot recover with all the data in flight rapidly enough (particularly in the case of streaming video), all the lost data needs to be resent, and you get a collapse and a tcp reset. And then the process starts again. The more bandwidth you are using up, the easier (and faster ) it is to trig= ger. tcp's response time vs buffering induced latency is quadratic. if you have 10x more buffering in the path than needed, it takes 100x longer to recover (so your recovery time went from ~10ms to 1000) We're seeing buffering all over the internet in multiple types of devices well in excess of that. Your buffers don't always fill, as some people have sufficient bandwidth and correctly operating gear. Most importantly very few people try to send sustained bursts of tcp data longer than a few seconds, which is why this is normally so hard to see. You were streaming 5Mbit which is far more than just about anybody.... Second most importantly it's a problem that is hard to trigger at sub ms rtts (like what you would get on local ethernet during test) , but gets rapidly easier as your rtts > 1 and the randomness of the internet kicks in. For WAY more detail on bufferbloat the cacm articles are canonical. http://cacm.acm.org/magazines/2012/1/144810-bufferbloat/fulltext See chart 4B in particular for a graphical explanation of how window size and rtts are interrelated. Your current setting now is overbuffered, but far less massively so. With the short rtts the quadratic-isms kick in, but limiting the send window size is still sub-optimal. I'm very sorry your show hit this problem, so hard, and that it took so long to figure out. It will make a great case study, and I would love it of a few of the amazing graphics and sound talents you had at "the gathering" - could lend their vision to explaining what bufferbloat is! jg's videos are helpful but they don't rock out to techno! nor do they feature led-lit juggling balls (wow!), nor spectral snakes that dissolve into visual madness. I am glad I had a chance to watch the udp stream and sad to know that so few others were able to enjoy the show. Perhaps using an rtp based streaming method, particularly over ipv6, will give you a better result next year. --=20 Dave T=E4ht SKYPE: davetaht US Tel: 1-239-829-5608 http://www.bufferbloat.net