From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-oi0-x235.google.com (mail-oi0-x235.google.com [IPv6:2607:f8b0:4003:c06::235]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by huchra.bufferbloat.net (Postfix) with ESMTPS id DAD7A21F282 for ; Tue, 21 Apr 2015 03:35:22 -0700 (PDT) Received: by oiko83 with SMTP id o83so148913972oik.1 for ; Tue, 21 Apr 2015 03:35:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:content-type; bh=FfV50hpcMTexTjCSzBFRTlAjfoRKQxO1kyu0RTjLsdE=; b=bAOQWhN7eyxfFr2cK5jWzqZ3kz2kVJGGuDSwkY1StsrGk3cHSpDkkzlQqRdT6UawXQ bVdmXUFaW1U3Z6hnXDJ+eJLf+CmjQeEGMbxO753eUd8QYrSv7fGMrCNqgnIS3Sj1lg0m KT/HFeCrPPYH2CRxjcxMO5xf+wO3VSYW4IprdTty97TRQ9oUJ2G7a2MazeLmDozMS/VR 3XkTna2jDhtqs24oAAZ/YaJH3CVcmUUZTHohVsv0NCbRMdVe2C/bSwWbv3v2km2Beotl N9o/LLYHFuaJ22GXs9DJ6jO6U0GrLchDIzXoaYZBlt7mHAQJHZexulSFnvultmiyGUpa 2sSA== MIME-Version: 1.0 X-Received: by 10.60.112.65 with SMTP id io1mr18293721oeb.66.1429612521749; Tue, 21 Apr 2015 03:35:21 -0700 (PDT) Sender: justinbeech@gmail.com Received: by 10.76.101.132 with HTTP; Tue, 21 Apr 2015 03:35:21 -0700 (PDT) In-Reply-To: References: <75C1DDFF-FBD2-4825-A167-92DFCF6A7713@gmail.com> <8AD4493E-EA21-496D-923D-B4257B078A1C@gmx.de> <8E4F61CA-4274-4414-B4C0-F582167D66D6@gmx.de> <2C987A4B-7459-43C1-A49C-72F600776B00@gmail.com> <14cd9e74e48.27f7.e972a4f4d859b00521b2b659602cb2f9@superduper.net> Date: Tue, 21 Apr 2015 20:35:21 +1000 X-Google-Sender-Auth: sNx8PntP108cbkjUHzLPHaMuxd4 Message-ID: From: jb To: bloat Content-Type: multipart/alternative; boundary=001a1130d2fa8682040514399a9d Subject: Re: [Bloat] DSLReports Speed Test has latency measurement built-in X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 21 Apr 2015 10:35:51 -0000 --001a1130d2fa8682040514399a9d Content-Type: text/plain; charset=UTF-8 As I understand it (I thought) SO_SNDBUF and SO_RCVBUF are socket buffers for the application layer, they do not change the TCP window size either send or receive. Which is perhaps why they aren't used much. They don't do much good in iperf that's for sure! Might be wrong, but I agree with the premise - auto-tuning should work. Regarding my own equipment, I've seen a 2012 topic about the Billion 7800N I have, complaining it has buffer bloat. The replies to the topic suggested using QOS to get round the problem of uploading blowing up the latency sky high. Unfortunately it is a very popular and well regarded DSL modem at least in Australia AND cannot be flashed with dd-wrt or anything. So I think for me personally (and for people who use our speed test and complain about very choppy results on upload), this is the explanation I'll be giving: experiment with your gear at home, it'll be the problem. Currently the servers are running at a low maximum receive window. I'll be switching them back in a day, after I let this one guy witness the improvement it make for his connection. He has been at me for days saying the test has an issue because the upload on his bonded 5mbit+5mbit channel is so choppy. thanks On Tue, Apr 21, 2015 at 7:37 PM, Jonathan Morton wrote: > I would explain it a bit differently to David. There are a lot of > interrelated components and concepts in TCP, and its sometimes hard to see > which ones are relevant in a given situation. > > The key insight though is that there are two windows which are maintained > by the sender and receiver respectively, and data can only be sent if it > fits into BOTH windows. The receive window is effectively set by that > sysctl, and the congestion window (maintained by the sender) is the one > that changes dynamically. > > The correct size of both windows is the bandwidth delay product of the > path between the two hosts. However, this size varies, so you can't set a > single size which works in all our even most situations. The general > approach that has the best chance of working is to set the receive window > large and rely on the congestion window to adapt. > > Incidentally, 200ms at say 2Mbps gives a BDP of about 40KB. > > The problem with that is that in most networks today, there is > insufficient information for the congestion window to find its ideal size. > It will grow until it receives an unambiguous congestion signal, typically > a lost packet or ECN flag. But that will most likely occur on queue > overflow at the bottleneck, and due to the resulting induced delay, the > sender will have been overdosing that queue for a while before it gets the > signal to back off - so probably a whole bunch of packets got lost in the > meantime. Then, after transmitting the lost packets, the sender has to wait > for the receiver to catch up with the smaller congestion window before it > can resume. > > Meanwhile, the receiver can't deliver any of the data it's receiving > because the lost packets belong in front of it. If you've ever noticed a > transfer that seems to stall and then suddenly catch up, that's due to a > lost packet and retransmission. The effect is known as "head of line > blocking", and can be used to detect packet loss at the application layer. > > Ironically, most hardware designers will tell you that buffers are meant > to smooth data delivery. It's true, but only when it doesn't overflow - and > TCP will always overflow a dumb queue if allowed to. > > Reducing the receive window, to a value below the native BDP of the path > plus the bottleneck queue length, can be used as a crude way to prevent the > bottleneck queue from overflowing. Then, the congestion window will grow to > the receive window size and stay there, and TCP will enter a steady state > where every ack results in the next packet(s) being sent. (Most receivers > won't send an ack for every received packet, as long as none are missing.) > > However, running multiple flows in parallel using a receive window tuned > for one flow will double the data in flight, and the queue may once again > overflow. If you look only at aggregate throughput, you might not notice > this because parallel TCPs tend to fill in each others' gaps. But the > individual flow throughputs will show the same "head of line blocking" > effect. > > - Jonathan Morton > --001a1130d2fa8682040514399a9d Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
As I understand it (I thoug= ht) SO_SNDBUF and SO_RCVBUF are socket buffers for the application layer, t= hey do not change the TCP window size either send or receive. Which is perh= aps why they aren't used much. They don't do much good in iperf tha= t's for sure!=C2=A0Might be wrong= , but I agree with the premise - auto-tuning should work.

Regarding my own equipment, = I've seen a 2012 topic about the Billion 7800N I have, complaining it h= as buffer bloat. The replies to the topic suggested using QOS to get round = the problem of uploading blowing up the latency sky high. Unfortunately it = is a very popular and well regarded DSL modem at least in Australia AND can= not be flashed with dd-wrt or anything. So I think for me personally (and f= or people who use our speed test and complain about very choppy results on = upload), this is the explanation I'll be giving: experiment with your g= ear at home, it'll be the problem.

Currently t= he servers are running at a low maximum receive window. I'll be switchi= ng them back in a day, after I let this one guy witness the improvement it = make for his connection. He has been at me for days saying the test has an = issue because the upload on his bonded 5mbit+5mbit channel is so choppy.

thanks


On Tue, Apr 21, 2015 at 7:37 PM, Jon= athan Morton <chromatix99@gmail.com> wrote:

I would explain it a bit differently t= o David. There are a lot of interrelated components and concepts in TCP, an= d its sometimes hard to see which ones are relevant in a given situation.

The key insight though is that there are two windows which a= re maintained by the sender and receiver respectively, and data can only be= sent if it fits into BOTH windows. The receive window is effectively set b= y that sysctl, and the congestion window (maintained by the sender) is the = one that changes dynamically.

The correct size of both windows is the bandwidth delay prod= uct of the path between the two hosts. However, this size varies, so you ca= n't set a single size which works in all our even most situations. The = general approach that has the best chance of working is to set the receive = window large and rely on the congestion window to adapt.

Incidentally, 200ms at say 2Mbps gives a BDP of about 40KB.<= /p>

The problem with that is that in most networks today, there = is insufficient information for the congestion window to find its ideal siz= e. It will grow until it receives an unambiguous congestion signal, typical= ly a lost packet or ECN flag. But that will most likely occur on queue over= flow at the bottleneck, and due to the resulting induced delay, the sender = will have been overdosing that queue for a while before it gets the signal = to back off - so probably a whole bunch of packets got lost in the meantime= . Then, after transmitting the lost packets, the sender has to wait for the= receiver to catch up with the smaller congestion window before it can resu= me.

Meanwhile, the receiver can't deliver any of the data it= 's receiving because the lost packets belong in front of it. If you'= ;ve ever noticed a transfer that seems to stall and then suddenly catch up,= that's due to a lost packet and retransmission. The effect is known as= "head of line blocking", and can be used to detect packet loss a= t the application layer.

Ironically, most hardware designers will tell you that buffe= rs are meant to smooth data delivery. It's true, but only when it doesn= 't overflow - and TCP will always overflow a dumb queue if allowed to.<= /p>

Reducing the receive window, to a value below the native BDP= of the path plus the bottleneck queue length, can be used as a crude way t= o prevent the bottleneck queue from overflowing. Then, the congestion windo= w will grow to the receive window size and stay there, and TCP will enter a= steady state where every ack results in the next packet(s) being sent. (Mo= st receivers won't send an ack for every received packet, as long as no= ne are missing.)

However, running multiple flows in parallel using a receive = window tuned for one flow will double the data in flight, and the queue may= once again overflow. If you look only at aggregate throughput, you might n= ot notice this because parallel TCPs tend to fill in each others' gaps.= But the individual flow throughputs will show the same "head of line = blocking" effect.

- Jonathan Morton


--001a1130d2fa8682040514399a9d--