From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ie0-x22a.google.com (mail-ie0-x22a.google.com [IPv6:2607:f8b0:4001:c03::22a]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by huchra.bufferbloat.net (Postfix) with ESMTPS id 7E17921F2AE for ; Tue, 21 Apr 2015 00:35:33 -0700 (PDT) Received: by iecrt8 with SMTP id rt8so7793879iec.0 for ; Tue, 21 Apr 2015 00:35:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:content-type; bh=R7MfpUinBU4EbOdf3QUmKaF324mVvwqq88BWKm5m8nY=; b=cgdgM22gMsxArwnTCAXG2Hqp/lS+eCCw4evpUze3qSobfYshfDM5rPpzkNw2ogVq// Vy90P5Ytze04M+jEDEUArjfyJUL2LgEd2akNj0sf7aB/dn/2T2pqH6IS55jRElJ6p8Sk TdoVaHdShTP/1yDySJhFz2ePsJ6CtaZaKnHuZpEBTIqjhPSlwoch0ytmDaqMYDMDgkyG ObloG0FIM8b5z8JckXitLXtTavVbSsbGShB5UMCB3WqgcoTkEzkO/fhm+CFFXUNoSnZp 8YIOAVXjXLAPxP8x8/RiMAg4fWYpoyVnSaAYh0ECLk+R7fvHFwBIyWuHkd4XPPqy5ruA uuxw== MIME-Version: 1.0 X-Received: by 10.50.64.244 with SMTP id r20mr26483332igs.48.1429601732290; Tue, 21 Apr 2015 00:35:32 -0700 (PDT) Sender: justinbeech@gmail.com Received: by 10.50.107.42 with HTTP; Tue, 21 Apr 2015 00:35:32 -0700 (PDT) In-Reply-To: References: <75C1DDFF-FBD2-4825-A167-92DFCF6A7713@gmail.com> <8AD4493E-EA21-496D-923D-B4257B078A1C@gmx.de> <8E4F61CA-4274-4414-B4C0-F582167D66D6@gmx.de> <2C987A4B-7459-43C1-A49C-72F600776B00@gmail.com> <14cd9e74e48.27f7.e972a4f4d859b00521b2b659602cb2f9@superduper.net> Date: Tue, 21 Apr 2015 17:35:32 +1000 X-Google-Sender-Auth: yVwqgRDLh7yLfzMmJMYAolGOb3M Message-ID: From: jb To: David Lang , bloat Content-Type: multipart/alternative; boundary=047d7bea3d226c6d74051437171c Subject: Re: [Bloat] DSLReports Speed Test has latency measurement built-in X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 21 Apr 2015 07:36:01 -0000 --047d7bea3d226c6d74051437171c Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable > the receiver advertizes a large receive window, so the sender doesn't pause > until there is that much data outstanding, or they get a timeout of a packet as > a signal to slow down. > and because you have a gig-E link locally, your machine generates traffic \ > very rapidly, until all that data is 'in flight'. but it's really sitting in the buffer of > router trying to get through. Hmm, then I have a quandary because I can easily solve the nasty bumpy upload graphs by keeping the advertised receive window on the server capped low, however then, paradoxically, there is no more sign of buffer bloat in the result, at least for the upload phase. (The graph under the upload/download graphs for my results shows almost no latency increase during the upload phase, now). Or, I can crank it back open again, serving people with fiber connections without having to run heaps of streams in parallel -- and then have people complain that the upload result is inefficient, or bumpy, vs what they expect. And I can't offer an option, because the server receive window (I think) cannot be set on a case by case basis. You set it for all TCP and forget it= . I suspect you guys are going to say the server should be left with a large max receive window.. and let people complain to find out what their issue is. BTW my setup is wire to billion 7800N, which is a DSL modem and router. I believe it is a linux based (judging from the system log) device. cheers, -Justin On Tue, Apr 21, 2015 at 2:47 PM, David Lang wrote: > On Tue, 21 Apr 2015, jb wrote: > > I've discovered something perhaps you guys can explain it better or shed >> some light. >> It isn't specifically to do with buffer bloat but it is to do with TCP >> tuning. >> >> Attached is two pictures of my upload to New York speed test server with= 1 >> stream. >> It doesn't make any difference if it is 1 stream or 8 streams, the pictu= re >> and behaviour remains the same. >> I am 200ms from new york so it qualifies as a fairly long (but not very >> fat) pipe. >> >> The nice smooth one is with linux tcp_rmem set to '4096 32768 65535' (on >> the server) >> The ugly bumpy one is with linux tcp_rmem set to '4096 65535 67108864' (= on >> the server) >> >> It actually doesn't matter what that last huge number is, once it goes >> much >> about 65k, e.g. 128k or 256k or beyond things get bumpy and ugly on the >> upload speed. >> >> Now as I understand this setting, it is the tcp receive window that Linu= x >> advertises, and the last number sets the maximum size it can get to (for >> one TCP stream). >> >> For users with very fast upload speeds, they do not see an ugly bumpy >> upload graph, it is smooth and sustained. >> But for the majority of users (like me) with uploads less than 5 to >> 10mbit, >> we frequently see the ugly graph. >> >> The second tcp_rmem setting is how I have been running the speed test >> servers. >> >> Up to now I thought this was just the distance of the speedtest from the >> interface: perhaps the browser was buffering a lot, and didn't feed back >> progress but now I realise the bumpy one is actually being influenced by >> the server receive window. >> >> I guess my question is this: Why does ALLOWING a large receive window >> appear to encourage problems with upload smoothness?? >> >> This implies that setting the receive window should be done on a >> connection >> by connection basis: small for slow connections, large, for high speed, >> long distance connections. >> > > This is classic bufferbloat > > the receiver advertizes a large receive window, so the sender doesn't > pause until there is that much data outstanding, or they get a timeout of= a > packet as a signal to slow down. > > and because you have a gig-E link locally, your machine generates traffic > very rapidly, until all that data is 'in flight'. but it's really sitting > in the buffer of a router trying to get through. > > then when a packet times out, the sender slows down a smidge and > retransmits it. But the old packet is still sitting in a queue, eating > bandwidth. the packets behind it are also going to timeout and be > retransmitted before your first retransmitted packet gets through, so you > have a large slug of data that's being retransmitted, and the first of th= e > replacement data can't get through until the last of the old (timed out) > data is transmitted. > > then when data starts flowing again, the sender again tries to fill up th= e > window with data in flight. > > In addition, if I cap it to 65k, for reasons of smoothness, >> that means the bandwidth delay product will keep maximum speed per uploa= d >> stream quite low. So a symmetric or gigabit connection is going to need = a >> ton of parallel streams to see full speed. >> >> Most puzzling is why would anything special be required on the Client --= > >> Server side of the equation >> but nothing much appears wrong with the Server --> Client side, whether >> speeds are very low (GPRS) or very high (gigabit). >> > > but what window sizes are these clients advertising? > > > Note that also I am not yet sure if smoothness =3D=3D better throughput.= I >> have >> noticed upload speeds for some people often being under their claimed sy= nc >> rate by 10 or 20% but I've no logs that show the bumpy graph is showing >> inefficiency. Maybe. >> > > If you were to do a packet capture on the server side, you would see that > you have a bunch of packets that are arriving multiple times, but the fir= st > time "does't count" because the replacement is already on the way. > > so your overall throughput is lower for two reasons > > 1. it's bursty, and there are times when the connection actually is idle > (after you have a lot of timed out packets, the sender needs to ramp up > it's speed again) > > 2. you are sending some packets multiple times, consuming more total > bandwidth for the same 'goodput' (effective throughput) > > David Lang > > > help! >> >> >> On Tue, Apr 21, 2015 at 12:56 PM, Simon Barber >> wrote: >> >> One thing users understand is slow web access. Perhaps translating the >>> latency measurement into 'a typical web page will take X seconds longer >>> to >>> load', or even stating the impact as 'this latency causes a typical web >>> page to load slower, as if your connection was only YY% of the measured >>> speed.' >>> >>> Simon >>> >>> Sent with AquaMail for Android >>> http://www.aqua-mail.com >>> >>> >>> >>> On April 19, 2015 1:54:19 PM Jonathan Morton >>> wrote: >>> >>> >>>> Frequency readouts are probably more accessible to the latter. >>> >>>> >>>>>>> The frequency domain more accessible to laypersons? I have my >>>>>>> >>>>>> doubts ;) >>>> >>>>> >>>>>> Gamers, at least, are familiar with =E2=80=9Cframes per second=E2=80= =9D and how that >>>>>> >>>>> corresponds to their monitor=E2=80=99s refresh rate. >>>> >>>>> >>>>> I am sure they can easily transform back into time domain to ge= t >>>>> >>>> the frame period ;) . I am partly kidding, I think your idea is great >>>> in >>>> that it is a truly positive value which could lend itself to being use= d >>>> in >>>> ISP/router manufacturer advertising, and hence might work in the real >>>> work; >>>> on the other hand I like to keep data as =E2=80=9Craw=E2=80=9D as poss= ible (not that >>>> ^(-1) >>>> is a transformation worthy of being called data massage). >>>> >>>>> >>>>> The desirable range of latencies, when converted to Hz, happens to b= e >>>>>> >>>>> roughly the same as the range of desirable frame rates. >>>> >>>>> >>>>> Just to play devils advocate, the interesting part is time or >>>>> >>>> saving time so seconds or milliseconds are also intuitively >>>> understandable >>>> and can be easily added ;) >>>> >>>> Such readouts are certainly interesting to people like us. I have no >>>> objection to them being reported alongside a frequency readout. But I >>>> think most people are not interested in =E2=80=9Ctime savings=E2=80=9D= measured in >>>> milliseconds; they=E2=80=99re much more aware of the minute- and hour-= level time >>>> savings associated with greater bandwidth. >>>> >>>> - Jonathan Morton >>>> >>>> _______________________________________________ >>>> Bloat mailing list >>>> Bloat@lists.bufferbloat.net >>>> https://lists.bufferbloat.net/listinfo/bloat >>>> >>>> >>> >>> _______________________________________________ >>> Bloat mailing list >>> Bloat@lists.bufferbloat.net >>> https://lists.bufferbloat.net/listinfo/bloat >>> >>> > _______________________________________________ > Bloat mailing list > Bloat@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/bloat > > --047d7bea3d226c6d74051437171c Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
> the recei= ver advertizes a large receive window, so the sender doesn't pause >= until there is that much data outstanding, or they get a timeout of a pack= et as > a signal to slow down.

> and because you have a gig-E = link locally, your machine generates traffic =C2=A0\
>=C2=A0very rap= idly, until all that data is 'in flight'. but it's really sitti= ng in the buffer of
> router trying to get through.

Hmm, then I have a quandary because I can easily sol= ve the nasty bumpy upload graphs by keeping the advertised receive window o= n the server capped low, however then, paradoxically, there is no more sign= of buffer bloat in the result, at least for the upload phase.

(The graph u= nder the upload/download graphs for my results shows almost no latency incr= ease during the upload phase, now).

=
Or, I can crank it back open again, ser= ving people with fiber connections without having to run heaps of streams i= n parallel -- and then have people complain that the upload result is ineff= icient, or bumpy, vs what they expect.
<= br>
And I can't offer an option, bec= ause the server receive window (I think) cannot be set on a case by case ba= sis. You set it for all TCP and forget it.

I suspect you guys are going to = say the server should be left with a large max receive window.. and let peo= ple complain to find out what their issue is.

BTW my setup is wire to billi= on 7800N, which is a DSL modem and router. I believe it is a linux based (j= udging from the system log) device.

=
cheers,
-Justin

On Tue, Apr 21, 2015 at 2:47 PM, David Lang <david@lang.hm> wrote:
On Tue, 21 Apr 2015, jb wrote:

I've discovered something perhaps you guys can explain it better or she= d
some light.
It isn't specifically to do with buffer bloat but it is to do with TCP<= br> tuning.

Attached is two pictures of my upload to New York speed test server with 1<= br> stream.
It doesn't make any difference if it is 1 stream or 8 streams, the pict= ure
and behaviour remains the same.
I am 200ms from new york so it qualifies as a fairly long (but not very
fat) pipe.

The nice smooth one is with linux tcp_rmem set to '4096 32768 65535'= ; (on
the server)
The ugly bumpy one is with linux tcp_rmem set to '4096 65535 67108864&#= 39; (on
the server)

It actually doesn't matter what that last huge number is, once it goes = much
about 65k, e.g. 128k or 256k or beyond things get bumpy and ugly on the
upload speed.

Now as I understand this setting, it is the tcp receive window that Linux advertises, and the last number sets the maximum size it can get to (for one TCP stream).

For users with very fast upload speeds, they do not see an ugly bumpy
upload graph, it is smooth and sustained.
But for the majority of users (like me) with uploads less than 5 to 10mbit,=
we frequently see the ugly graph.

The second tcp_rmem setting is how I have been running the speed test
servers.

Up to now I thought this was just the distance of the speedtest from the interface: perhaps the browser was buffering a lot, and didn't feed bac= k
progress but now I realise the bumpy one is actually being influenced by the server receive window.

I guess my question is this: Why does ALLOWING a large receive window
appear to encourage problems with upload smoothness??

This implies that setting the receive window should be done on a connection=
by connection basis: small for slow connections, large, for high speed,
long distance connections.

This is classic bufferbloat

the receiver advertizes a large receive window, so the sender doesn't p= ause until there is that much data outstanding, or they get a timeout of a = packet as a signal to slow down.

and because you have a gig-E link locally, your machine generates traffic v= ery rapidly, until all that data is 'in flight'. but it's reall= y sitting in the buffer of a router trying to get through.

then when a packet times out, the sender slows down a smidge and retransmit= s it. But the old packet is still sitting in a queue, eating bandwidth. the= packets behind it are also going to timeout and be retransmitted before yo= ur first retransmitted packet gets through, so you have a large slug of dat= a that's being retransmitted, and the first of the replacement data can= 't get through until the last of the old (timed out) data is transmitte= d.

then when data starts flowing again, the sender again tries to fill up the = window with data in flight.

In addition, if I cap it to 65k, for reasons of smoothness,
that means the bandwidth delay product will keep maximum speed per upload stream quite low. So a symmetric or gigabit connection is going to need a ton of parallel streams to see full speed.

Most puzzling is why would anything special be required on the Client -->= ;
Server side of the equation
but nothing much appears wrong with the Server --> Client side, whether<= br> speeds are very low (GPRS) or very high (gigabit).

 but what window sizes are these clients advertising?


Note that also I am not yet sure if smoothness =3D=3D better throughput. I = have
noticed upload speeds for some people often being under their claimed sync<= br> rate by 10 or 20% but I've no logs that show the bumpy graph is showing=
inefficiency. Maybe.

 If you were to do a packet capture on the server side, you would see that y= ou have a bunch of packets that are arriving multiple times, but the first = time "does't count" because the replacement is already on the= way.

so your overall throughput is lower for two reasons

1. it's bursty, and there are times when the connection actually is idl= e (after you have a lot of timed out packets, the sender needs to ramp up i= t's speed again)

2. you are sending some packets multiple times, consuming more total bandwi= dth for the same 'goodput' (effective throughput)

David Lang


help!


On Tue, Apr 21, 2015 at 12:56 PM, Simon Barber <simon@superduper.net> wrote:

One thing users understand is slow web access.=C2=A0 Perhaps translating th= e
latency measurement into 'a typical web page will take X seconds longer= to
load', or even stating the impact as 'this latency causes a typical= web
page to load slower, as if your connection was only YY% of the measured
speed.'

Simon

Sent with AquaMail for Android
http://www.aqua-mail= .com



On April 19, 2015 1:54:19 PM Jonathan Morton <chromatix99@gmail.com>
wrote:

>>>> Frequency readouts are probably more accessible to the lat= ter.

=C2=A0 =C2=A0 The frequency domain more accessible to laypersons? I have my=
doubts ;)

Gamers, at least, are familiar with =E2=80=9Cframes per second=E2=80=9D and= how that
corresponds to their monitor=E2=80=99s refresh rate.

=C2=A0 =C2=A0 =C2=A0 I am sure they can easily transform back into time dom= ain to get
the frame period ;) .=C2=A0 I am partly kidding, I think your idea is great= in
that it is a truly positive value which could lend itself to being used in<= br> ISP/router manufacturer advertising, and hence might work in the real work;=
on the other hand I like to keep data as =E2=80=9Craw=E2=80=9D as possible = (not that ^(-1)
is a transformation worthy of being called data massage).

The desirable range of latencies, when converted to Hz, happens to be
roughly the same as the range of desirable frame rates.

=C2=A0 =C2=A0 =C2=A0 Just to play devils advocate, the interesting part is = time or
saving time so seconds or milliseconds are also intuitively understandable<= br> and can be easily added ;)

Such readouts are certainly interesting to people like us.=C2=A0 I have no<= br> objection to them being reported alongside a frequency readout.=C2=A0 But I=
think most people are not interested in =E2=80=9Ctime savings=E2=80=9D meas= ured in
milliseconds; they=E2=80=99re much more aware of the minute- and hour-level= time
savings associated with greater bandwidth.

=C2=A0- Jonathan Morton

_______________________________________________
Bloat mailing list
Bloat@list= s.bufferbloat.net
= https://lists.bufferbloat.net/listinfo/bloat



_______________________________________________
Bloat mailing list
Bloat@list= s.bufferbloat.net
= https://lists.bufferbloat.net/listinfo/bloat


_______________________________________________
Bloat mailing list
Bloat@lists.bufferbloat.net<= /a>
= https://lists.bufferbloat.net/listinfo/bloat


--047d7bea3d226c6d74051437171c--