From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from masada.superduper.net (unknown [IPv6:2001:ba8:1f1:f263::2]) (using TLSv1 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (Client did not present a certificate) by huchra.bufferbloat.net (Postfix) with ESMTPS id 1FC3621F35D for ; Wed, 22 Apr 2015 18:37:16 -0700 (PDT) Received: from 199-116-72-167.public.monkeybrains.net ([199.116.72.167] helo=[192.168.0.12]) by masada.superduper.net with esmtpsa (TLS1.0:RSA_ARCFOUR_MD5:128) (Exim 4.80) (envelope-from ) id 1Yl64k-0007Cb-3a; Thu, 23 Apr 2015 02:37:13 +0100 From: Simon Barber To: David Lang Date: Wed, 22 Apr 2015 18:37:07 -0700 Message-ID: <14ce3ebce00.27f7.e972a4f4d859b00521b2b659602cb2f9@superduper.net> In-Reply-To: References: <75C1DDFF-FBD2-4825-A167-92DFCF6A7713@gmail.com> <8AD4493E-EA21-496D-923D-B4257B078A1C@gmx.de> <8E4F61CA-4274-4414-B4C0-F582167D66D6@gmx.de> <2C987A4B-7459-43C1-A49C-72F600776B00@gmail.com> <14cd9e74e48.27f7.e972a4f4d859b00521b2b659602cb2f9@superduper.net> <14ce18b0a40.27f7.e972a4f4d859b00521b2b659602cb2f9@superduper.net> User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.6.0 AquaMail/1.5.0.19 (build: 2100846) MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: 8bit X-Spam-Score: -2.9 (--) Cc: bloat Subject: Re: [Bloat] DSLReports Speed Test has latency measurement built-in X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Apr 2015 01:37:45 -0000 Does this happen even with Sack? Simon Sent with AquaMail for Android http://www.aqua-mail.com On April 22, 2015 10:36:11 AM David Lang wrote: > Data that's received and not used doesn't really matter (a tree falls in the > woods type of thing). > > The head of line blocking can cause a chunk of packets to be retransmitted, > even > though the receiving machine got them the first time. So looking at the > received > bytes gives you a false picture of what is going on. > > David Lang > > On Wed, 22 Apr 2015, Simon Barber wrote: > > > The bumps are due to packet loss causing head of line blocking. Until the > > lost packet is retransmitted the receiver can't release any subsequent > > received packets to the application due to the requirement for in order > > delivery. If you counted received bytes with a packet counter rather than > > looking at application level you would be able to illustrate that data was > > being received smoothly (even though out of order). > > > > Simon > > > > Sent with AquaMail for Android > > http://www.aqua-mail.com > > > > > > On April 21, 2015 7:21:09 AM David Lang wrote: > > > >> On Tue, 21 Apr 2015, jb wrote: > >> > >> >> the receiver advertizes a large receive window, so the sender doesn't > >> > pause > until there is that much data outstanding, or they get a timeout > >> of > >> > a packet as > a signal to slow down. > >> > > >> >> and because you have a gig-E link locally, your machine generates > >> traffic > >> > \ > >> >> very rapidly, until all that data is 'in flight'. but it's really > >> sitting > >> > in the buffer of > >> >> router trying to get through. > >> > > >> > Hmm, then I have a quandary because I can easily solve the nasty bumpy > >> > upload graphs by keeping the advertised receive window on the server > >> capped > >> > low, however then, paradoxically, there is no more sign of buffer bloat > >> in > >> > the result, at least for the upload phase. > >> > > >> > (The graph under the upload/download graphs for my results shows almost > >> no > >> > latency increase during the upload phase, now). > >> > > >> > Or, I can crank it back open again, serving people with fiber connections > >> > without having to run heaps of streams in parallel -- and then have > >> people > >> > complain that the upload result is inefficient, or bumpy, vs what they > >> > expect. > >> > >> well, many people expect it to be bumpy (I've heard ISPs explain to > >> customers > >> that when a link is full it is bumpy, that's just the way things work) > >> > >> > And I can't offer an option, because the server receive window (I think) > >> > cannot be set on a case by case basis. You set it for all TCP and forget > >> it. > >> > >> I think you are right > >> > >> > I suspect you guys are going to say the server should be left with a > >> large > >> > max receive window.. and let people complain to find out what their issue > >> > is. > >> > >> what is your customer base? how important is it to provide faster service > >> to teh > >> fiber users? Are they transferring ISO images so the difference is > >> significant > >> to them? or are they downloading web pages where it's the difference > >> between a > >> half second and a quarter second? remember that you are seeing this on the > >> upload side. > >> > >> in the long run, fixing the problem at the client side is the best thing to > >> do, > >> but in the meantime, you sometimes have to work around broken customer > >> stuff. > >> > >> > BTW my setup is wire to billion 7800N, which is a DSL modem and router. I > >> > believe it is a linux based (judging from the system log) device. > >> > >> if it's linux based, it would be interesting to learn what sort of settings > >> it > >> has. It may be one of the rarer devices that has something in place already > >> to > >> do active queue management. > >> > >> David Lang > >> > >> > cheers, > >> > -Justin > >> > > >> > On Tue, Apr 21, 2015 at 2:47 PM, David Lang wrote: > >> > > >> >> On Tue, 21 Apr 2015, jb wrote: > >> >> > >> >> I've discovered something perhaps you guys can explain it better or > >> shed > >> >>> some light. > >> >>> It isn't specifically to do with buffer bloat but it is to do with TCP > >> >>> tuning. > >> >>> > >> >>> Attached is two pictures of my upload to New York speed test server > >> with 1 > >> >>> stream. > >> >>> It doesn't make any difference if it is 1 stream or 8 streams, the > >> picture > >> >>> and behaviour remains the same. > >> >>> I am 200ms from new york so it qualifies as a fairly long (but not very > >> >>> fat) pipe. > >> >>> > >> >>> The nice smooth one is with linux tcp_rmem set to '4096 32768 65535' > >> (on > >> >>> the server) > >> >>> The ugly bumpy one is with linux tcp_rmem set to '4096 65535 67108864' > >> (on > >> >>> the server) > >> >>> > >> >>> It actually doesn't matter what that last huge number is, once it goes > >> >>> much > >> >>> about 65k, e.g. 128k or 256k or beyond things get bumpy and ugly on the > >> >>> upload speed. > >> >>> > >> >>> Now as I understand this setting, it is the tcp receive window that > >> Linux > >> >>> advertises, and the last number sets the maximum size it can get to > >> (for > >> >>> one TCP stream). > >> >>> > >> >>> For users with very fast upload speeds, they do not see an ugly bumpy > >> >>> upload graph, it is smooth and sustained. > >> >>> But for the majority of users (like me) with uploads less than 5 to > >> >>> 10mbit, > >> >>> we frequently see the ugly graph. > >> >>> > >> >>> The second tcp_rmem setting is how I have been running the speed test > >> >>> servers. > >> >>> > >> >>> Up to now I thought this was just the distance of the speedtest from > >> the > >> >>> interface: perhaps the browser was buffering a lot, and didn't feed > >> back > >> >>> progress but now I realise the bumpy one is actually being influenced > >> by > >> >>> the server receive window. > >> >>> > >> >>> I guess my question is this: Why does ALLOWING a large receive window > >> >>> appear to encourage problems with upload smoothness?? > >> >>> > >> >>> This implies that setting the receive window should be done on a > >> >>> connection > >> >>> by connection basis: small for slow connections, large, for high speed, > >> >>> long distance connections. > >> >>> > >> >> > >> >> This is classic bufferbloat > >> >> > >> >> the receiver advertizes a large receive window, so the sender doesn't > >> >> pause until there is that much data outstanding, or they get a timeout > >> of a > >> >> packet as a signal to slow down. > >> >> > >> >> and because you have a gig-E link locally, your machine generates > >> traffic > >> >> very rapidly, until all that data is 'in flight'. but it's really > >> sitting > >> >> in the buffer of a router trying to get through. > >> >> > >> >> then when a packet times out, the sender slows down a smidge and > >> >> retransmits it. But the old packet is still sitting in a queue, eating > >> >> bandwidth. the packets behind it are also going to timeout and be > >> >> retransmitted before your first retransmitted packet gets through, so > >> you > >> >> have a large slug of data that's being retransmitted, and the first of > >> the > >> >> replacement data can't get through until the last of the old (timed out) > >> >> data is transmitted. > >> >> > >> >> then when data starts flowing again, the sender again tries to fill up > >> the > >> >> window with data in flight. > >> >> > >> >> In addition, if I cap it to 65k, for reasons of smoothness, > >> >>> that means the bandwidth delay product will keep maximum speed per > >> upload > >> >>> stream quite low. So a symmetric or gigabit connection is going to need > >> a > >> >>> ton of parallel streams to see full speed. > >> >>> > >> >>> Most puzzling is why would anything special be required on the Client > >> --> > >> >>> Server side of the equation > >> >>> but nothing much appears wrong with the Server --> Client side, whether > >> >>> speeds are very low (GPRS) or very high (gigabit). > >> >>> > >> >> > >> >> but what window sizes are these clients advertising? > >> >> > >> >> > >> >> Note that also I am not yet sure if smoothness == better throughput. I > >> >>> have > >> >>> noticed upload speeds for some people often being under their claimed > >> sync > >> >>> rate by 10 or 20% but I've no logs that show the bumpy graph is showing > >> >>> inefficiency. Maybe. > >> >>> > >> >> > >> >> If you were to do a packet capture on the server side, you would see > >> that > >> >> you have a bunch of packets that are arriving multiple times, but the > >> first > >> >> time "does't count" because the replacement is already on the way. > >> >> > >> >> so your overall throughput is lower for two reasons > >> >> > >> >> 1. it's bursty, and there are times when the connection actually is idle > >> >> (after you have a lot of timed out packets, the sender needs to ramp up > >> >> it's speed again) > >> >> > >> >> 2. you are sending some packets multiple times, consuming more total > >> >> bandwidth for the same 'goodput' (effective throughput) > >> >> > >> >> David Lang > >> >> > >> >> > >> >> help! > >> >>> > >> >>> > >> >>> On Tue, Apr 21, 2015 at 12:56 PM, Simon Barber > >> >>> wrote: > >> >>> > >> >>> One thing users understand is slow web access. Perhaps translating > >> the > >> >>>> latency measurement into 'a typical web page will take X seconds > >> longer > >> >>>> to > >> >>>> load', or even stating the impact as 'this latency causes a typical > >> web > >> >>>> page to load slower, as if your connection was only YY% of the > >> measured > >> >>>> speed.' > >> >>>> > >> >>>> Simon > >> >>>> > >> >>>> Sent with AquaMail for Android > >> >>>> http://www.aqua-mail.com > >> >>>> > >> >>>> > >> >>>> > >> >>>> On April 19, 2015 1:54:19 PM Jonathan Morton > >> >>>> wrote: > >> >>>> > >> >>>>>>>> Frequency readouts are probably more accessible to the latter. > >> >>>> > >> >>>>> > >> >>>>>>>> The frequency domain more accessible to laypersons? I have my > >> >>>>>>>> > >> >>>>>>> doubts ;) > >> >>>>> > >> >>>>>> > >> >>>>>>> Gamers, at least, are familiar with “frames per second” and how > >> that > >> >>>>>>> > >> >>>>>> corresponds to their monitor’s refresh rate. > >> >>>>> > >> >>>>>> > >> >>>>>> I am sure they can easily transform back into time domain to > >> get > >> >>>>>> > >> >>>>> the frame period ;) . I am partly kidding, I think your idea is > >> great > >> >>>>> in > >> >>>>> that it is a truly positive value which could lend itself to being > >> used > >> >>>>> in > >> >>>>> ISP/router manufacturer advertising, and hence might work in the real > >> >>>>> work; > >> >>>>> on the other hand I like to keep data as “raw” as possible (not that > >> >>>>> ^(-1) > >> >>>>> is a transformation worthy of being called data massage). > >> >>>>> > >> >>>>>> > >> >>>>>> The desirable range of latencies, when converted to Hz, happens to > >> be > >> >>>>>>> > >> >>>>>> roughly the same as the range of desirable frame rates. > >> >>>>> > >> >>>>>> > >> >>>>>> Just to play devils advocate, the interesting part is time or > >> >>>>>> > >> >>>>> saving time so seconds or milliseconds are also intuitively > >> >>>>> understandable > >> >>>>> and can be easily added ;) > >> >>>>> > >> >>>>> Such readouts are certainly interesting to people like us. I have no > >> >>>>> objection to them being reported alongside a frequency readout. But > >> I > >> >>>>> think most people are not interested in “time savings” measured in > >> >>>>> milliseconds; they’re much more aware of the minute- and hour-level > >> time > >> >>>>> savings associated with greater bandwidth. > >> >>>>> > >> >>>>> - Jonathan Morton > >> >>>>> > >> >>>>> _______________________________________________ > >> >>>>> Bloat mailing list > >> >>>>> Bloat@lists.bufferbloat.net > >> >>>>> https://lists.bufferbloat.net/listinfo/bloat > >> >>>>> > >> >>>>> > >> >>>> > >> >>>> _______________________________________________ > >> >>>> Bloat mailing list > >> >>>> Bloat@lists.bufferbloat.net > >> >>>> https://lists.bufferbloat.net/listinfo/bloat > >> >>>> > >> >>>> > >> >> _______________________________________________ > >> >> Bloat mailing list > >> >> Bloat@lists.bufferbloat.net > >> >> https://lists.bufferbloat.net/listinfo/bloat > >> >> > >> >> > >> > > >> > >> > >> ---------- > >> _______________________________________________ > >> Bloat mailing list > >> Bloat@lists.bufferbloat.net > >> https://lists.bufferbloat.net/listinfo/bloat > >> > > > > > >