From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from bifrost.lang.hm (mail.lang.hm [64.81.33.126]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by huchra.bufferbloat.net (Postfix) with ESMTPS id DEB0821F32E for ; Wed, 22 Apr 2015 10:35:33 -0700 (PDT) Received: from asgard.lang.hm (asgard.lang.hm [10.0.0.100]) by bifrost.lang.hm (8.13.4/8.13.4/Debian-3) with ESMTP id t3MHZNnV028253; Wed, 22 Apr 2015 10:35:23 -0700 Date: Wed, 22 Apr 2015 10:35:23 -0700 (PDT) From: David Lang X-X-Sender: dlang@asgard.lang.hm To: Simon Barber In-Reply-To: <14ce18b0a40.27f7.e972a4f4d859b00521b2b659602cb2f9@superduper.net> Message-ID: References: <75C1DDFF-FBD2-4825-A167-92DFCF6A7713@gmail.com> <8AD4493E-EA21-496D-923D-B4257B078A1C@gmx.de> <8E4F61CA-4274-4414-B4C0-F582167D66D6@gmx.de> <2C987A4B-7459-43C1-A49C-72F600776B00@gmail.com> <14cd9e74e48.27f7.e972a4f4d859b00521b2b659602cb2f9@superduper.net> <14ce18b0a40.27f7.e972a4f4d859b00521b2b659602cb2f9@superduper.net> User-Agent: Alpine 2.02 (DEB 1266 2009-07-14) MIME-Version: 1.0 Content-Type: MULTIPART/MIXED; BOUNDARY="680960-1659562583-1429724123=:1097" Cc: bloat Subject: Re: [Bloat] DSLReports Speed Test has latency measurement built-in X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Apr 2015 17:36:02 -0000 This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. --680960-1659562583-1429724123=:1097 Content-Type: TEXT/PLAIN; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8BIT Data that's received and not used doesn't really matter (a tree falls in the woods type of thing). The head of line blocking can cause a chunk of packets to be retransmitted, even though the receiving machine got them the first time. So looking at the received bytes gives you a false picture of what is going on. David Lang On Wed, 22 Apr 2015, Simon Barber wrote: > The bumps are due to packet loss causing head of line blocking. Until the > lost packet is retransmitted the receiver can't release any subsequent > received packets to the application due to the requirement for in order > delivery. If you counted received bytes with a packet counter rather than > looking at application level you would be able to illustrate that data was > being received smoothly (even though out of order). > > Simon > > Sent with AquaMail for Android > http://www.aqua-mail.com > > > On April 21, 2015 7:21:09 AM David Lang wrote: > >> On Tue, 21 Apr 2015, jb wrote: >> >> >> the receiver advertizes a large receive window, so the sender doesn't >> > pause > until there is that much data outstanding, or they get a timeout >> of >> > a packet as > a signal to slow down. >> > >> >> and because you have a gig-E link locally, your machine generates >> traffic >> > \ >> >> very rapidly, until all that data is 'in flight'. but it's really >> sitting >> > in the buffer of >> >> router trying to get through. >> > >> > Hmm, then I have a quandary because I can easily solve the nasty bumpy >> > upload graphs by keeping the advertised receive window on the server >> capped >> > low, however then, paradoxically, there is no more sign of buffer bloat >> in >> > the result, at least for the upload phase. >> > >> > (The graph under the upload/download graphs for my results shows almost >> no >> > latency increase during the upload phase, now). >> > >> > Or, I can crank it back open again, serving people with fiber connections >> > without having to run heaps of streams in parallel -- and then have >> people >> > complain that the upload result is inefficient, or bumpy, vs what they >> > expect. >> >> well, many people expect it to be bumpy (I've heard ISPs explain to >> customers >> that when a link is full it is bumpy, that's just the way things work) >> >> > And I can't offer an option, because the server receive window (I think) >> > cannot be set on a case by case basis. You set it for all TCP and forget >> it. >> >> I think you are right >> >> > I suspect you guys are going to say the server should be left with a >> large >> > max receive window.. and let people complain to find out what their issue >> > is. >> >> what is your customer base? how important is it to provide faster service >> to teh >> fiber users? Are they transferring ISO images so the difference is >> significant >> to them? or are they downloading web pages where it's the difference >> between a >> half second and a quarter second? remember that you are seeing this on the >> upload side. >> >> in the long run, fixing the problem at the client side is the best thing to >> do, >> but in the meantime, you sometimes have to work around broken customer >> stuff. >> >> > BTW my setup is wire to billion 7800N, which is a DSL modem and router. I >> > believe it is a linux based (judging from the system log) device. >> >> if it's linux based, it would be interesting to learn what sort of settings >> it >> has. It may be one of the rarer devices that has something in place already >> to >> do active queue management. >> >> David Lang >> >> > cheers, >> > -Justin >> > >> > On Tue, Apr 21, 2015 at 2:47 PM, David Lang wrote: >> > >> >> On Tue, 21 Apr 2015, jb wrote: >> >> >> >> I've discovered something perhaps you guys can explain it better or >> shed >> >>> some light. >> >>> It isn't specifically to do with buffer bloat but it is to do with TCP >> >>> tuning. >> >>> >> >>> Attached is two pictures of my upload to New York speed test server >> with 1 >> >>> stream. >> >>> It doesn't make any difference if it is 1 stream or 8 streams, the >> picture >> >>> and behaviour remains the same. >> >>> I am 200ms from new york so it qualifies as a fairly long (but not very >> >>> fat) pipe. >> >>> >> >>> The nice smooth one is with linux tcp_rmem set to '4096 32768 65535' >> (on >> >>> the server) >> >>> The ugly bumpy one is with linux tcp_rmem set to '4096 65535 67108864' >> (on >> >>> the server) >> >>> >> >>> It actually doesn't matter what that last huge number is, once it goes >> >>> much >> >>> about 65k, e.g. 128k or 256k or beyond things get bumpy and ugly on the >> >>> upload speed. >> >>> >> >>> Now as I understand this setting, it is the tcp receive window that >> Linux >> >>> advertises, and the last number sets the maximum size it can get to >> (for >> >>> one TCP stream). >> >>> >> >>> For users with very fast upload speeds, they do not see an ugly bumpy >> >>> upload graph, it is smooth and sustained. >> >>> But for the majority of users (like me) with uploads less than 5 to >> >>> 10mbit, >> >>> we frequently see the ugly graph. >> >>> >> >>> The second tcp_rmem setting is how I have been running the speed test >> >>> servers. >> >>> >> >>> Up to now I thought this was just the distance of the speedtest from >> the >> >>> interface: perhaps the browser was buffering a lot, and didn't feed >> back >> >>> progress but now I realise the bumpy one is actually being influenced >> by >> >>> the server receive window. >> >>> >> >>> I guess my question is this: Why does ALLOWING a large receive window >> >>> appear to encourage problems with upload smoothness?? >> >>> >> >>> This implies that setting the receive window should be done on a >> >>> connection >> >>> by connection basis: small for slow connections, large, for high speed, >> >>> long distance connections. >> >>> >> >> >> >> This is classic bufferbloat >> >> >> >> the receiver advertizes a large receive window, so the sender doesn't >> >> pause until there is that much data outstanding, or they get a timeout >> of a >> >> packet as a signal to slow down. >> >> >> >> and because you have a gig-E link locally, your machine generates >> traffic >> >> very rapidly, until all that data is 'in flight'. but it's really >> sitting >> >> in the buffer of a router trying to get through. >> >> >> >> then when a packet times out, the sender slows down a smidge and >> >> retransmits it. But the old packet is still sitting in a queue, eating >> >> bandwidth. the packets behind it are also going to timeout and be >> >> retransmitted before your first retransmitted packet gets through, so >> you >> >> have a large slug of data that's being retransmitted, and the first of >> the >> >> replacement data can't get through until the last of the old (timed out) >> >> data is transmitted. >> >> >> >> then when data starts flowing again, the sender again tries to fill up >> the >> >> window with data in flight. >> >> >> >> In addition, if I cap it to 65k, for reasons of smoothness, >> >>> that means the bandwidth delay product will keep maximum speed per >> upload >> >>> stream quite low. So a symmetric or gigabit connection is going to need >> a >> >>> ton of parallel streams to see full speed. >> >>> >> >>> Most puzzling is why would anything special be required on the Client >> --> >> >>> Server side of the equation >> >>> but nothing much appears wrong with the Server --> Client side, whether >> >>> speeds are very low (GPRS) or very high (gigabit). >> >>> >> >> >> >> but what window sizes are these clients advertising? >> >> >> >> >> >> Note that also I am not yet sure if smoothness == better throughput. I >> >>> have >> >>> noticed upload speeds for some people often being under their claimed >> sync >> >>> rate by 10 or 20% but I've no logs that show the bumpy graph is showing >> >>> inefficiency. Maybe. >> >>> >> >> >> >> If you were to do a packet capture on the server side, you would see >> that >> >> you have a bunch of packets that are arriving multiple times, but the >> first >> >> time "does't count" because the replacement is already on the way. >> >> >> >> so your overall throughput is lower for two reasons >> >> >> >> 1. it's bursty, and there are times when the connection actually is idle >> >> (after you have a lot of timed out packets, the sender needs to ramp up >> >> it's speed again) >> >> >> >> 2. you are sending some packets multiple times, consuming more total >> >> bandwidth for the same 'goodput' (effective throughput) >> >> >> >> David Lang >> >> >> >> >> >> help! >> >>> >> >>> >> >>> On Tue, Apr 21, 2015 at 12:56 PM, Simon Barber >> >>> wrote: >> >>> >> >>> One thing users understand is slow web access. Perhaps translating >> the >> >>>> latency measurement into 'a typical web page will take X seconds >> longer >> >>>> to >> >>>> load', or even stating the impact as 'this latency causes a typical >> web >> >>>> page to load slower, as if your connection was only YY% of the >> measured >> >>>> speed.' >> >>>> >> >>>> Simon >> >>>> >> >>>> Sent with AquaMail for Android >> >>>> http://www.aqua-mail.com >> >>>> >> >>>> >> >>>> >> >>>> On April 19, 2015 1:54:19 PM Jonathan Morton >> >>>> wrote: >> >>>> >> >>>>>>>> Frequency readouts are probably more accessible to the latter. >> >>>> >> >>>>> >> >>>>>>>> The frequency domain more accessible to laypersons? I have my >> >>>>>>>> >> >>>>>>> doubts ;) >> >>>>> >> >>>>>> >> >>>>>>> Gamers, at least, are familiar with “frames per second” and how >> that >> >>>>>>> >> >>>>>> corresponds to their monitor’s refresh rate. >> >>>>> >> >>>>>> >> >>>>>> I am sure they can easily transform back into time domain to >> get >> >>>>>> >> >>>>> the frame period ;) . I am partly kidding, I think your idea is >> great >> >>>>> in >> >>>>> that it is a truly positive value which could lend itself to being >> used >> >>>>> in >> >>>>> ISP/router manufacturer advertising, and hence might work in the real >> >>>>> work; >> >>>>> on the other hand I like to keep data as “raw” as possible (not that >> >>>>> ^(-1) >> >>>>> is a transformation worthy of being called data massage). >> >>>>> >> >>>>>> >> >>>>>> The desirable range of latencies, when converted to Hz, happens to >> be >> >>>>>>> >> >>>>>> roughly the same as the range of desirable frame rates. >> >>>>> >> >>>>>> >> >>>>>> Just to play devils advocate, the interesting part is time or >> >>>>>> >> >>>>> saving time so seconds or milliseconds are also intuitively >> >>>>> understandable >> >>>>> and can be easily added ;) >> >>>>> >> >>>>> Such readouts are certainly interesting to people like us. I have no >> >>>>> objection to them being reported alongside a frequency readout. But >> I >> >>>>> think most people are not interested in “time savings” measured in >> >>>>> milliseconds; they’re much more aware of the minute- and hour-level >> time >> >>>>> savings associated with greater bandwidth. >> >>>>> >> >>>>> - Jonathan Morton >> >>>>> >> >>>>> _______________________________________________ >> >>>>> Bloat mailing list >> >>>>> Bloat@lists.bufferbloat.net >> >>>>> https://lists.bufferbloat.net/listinfo/bloat >> >>>>> >> >>>>> >> >>>> >> >>>> _______________________________________________ >> >>>> Bloat mailing list >> >>>> Bloat@lists.bufferbloat.net >> >>>> https://lists.bufferbloat.net/listinfo/bloat >> >>>> >> >>>> >> >> _______________________________________________ >> >> Bloat mailing list >> >> Bloat@lists.bufferbloat.net >> >> https://lists.bufferbloat.net/listinfo/bloat >> >> >> >> >> > >> >> >> ---------- >> _______________________________________________ >> Bloat mailing list >> Bloat@lists.bufferbloat.net >> https://lists.bufferbloat.net/listinfo/bloat >> > > > --680960-1659562583-1429724123=:1097--