From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-yk0-x244.google.com (mail-yk0-x244.google.com [IPv6:2607:f8b0:4002:c07::244]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by huchra.bufferbloat.net (Postfix) with ESMTPS id 3490721F3EF for ; Fri, 12 Jun 2015 11:44:34 -0700 (PDT) Received: by ykq19 with SMTP id 19so2324619ykq.1 for ; Fri, 12 Jun 2015 11:44:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=bnBMf/mb80tUX/UYq6+MFvTrkTSHrgo0hFBncqPWfMw=; b=Y+O4AmuesKg3vTFhS8nSeqznW2fXG9gDuP2UNYwL2HTJFA44viSK/RMxy8kcUiBz++ 1v5J9341UR80YYMoGqab3RMQx/stNVSdokT5z235kW5KegMCklPuTl28iXfd7ZMNfjME JTC6hFj10Q5nvm3uloYMgVJTQzfAF+FOY9o8G/N6gsV23EjjrmLUk+PJsH1mF505NAXl eVzjKFFr8t0aR30lsZ+q1Ozos8WVHWSBZTg7TS1IGUff6tPiOyru6+6YCOLlCepsimWH 6KowblmA+ZcPSnZrpF76mjaQIM5L5EW8VyFb6IBn4eZQf0uYju0C1anDdDXIMpejLIMi Ij8w== MIME-Version: 1.0 X-Received: by 10.129.106.133 with SMTP id f127mr20966719ywc.8.1434134673898; Fri, 12 Jun 2015 11:44:33 -0700 (PDT) Received: by 10.129.148.194 with HTTP; Fri, 12 Jun 2015 11:44:33 -0700 (PDT) In-Reply-To: References: <6AD8E150-5751-43AC-8F6C-8175C1E92DE1@gmx.de> Date: Fri, 12 Jun 2015 13:44:33 -0500 Message-ID: From: Benjamin Cronce To: Sebastian Moeller Content-Type: multipart/alternative; boundary=001a1149424ccc43970518567fea Cc: bloat Subject: Re: [Bloat] Bloat done correctly? X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Jun 2015 18:45:03 -0000 --001a1149424ccc43970518567fea Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable > Hi Benjamin, > > On Jun 12, 2015, at 17:33 , Benjamin Cronce wrote: > > > > On Fri, Jun 12, 2015 at 4:08 AM, Sebastian Moeller wrote: > > > Hi Benjamin, > > > > > > To go off onto a tangent: > > > > > > On Jun 12, 2015, at 06:45 , Benjamin Cronce wrote: > > > > > > > [...] > > > > Under load while doing P2P(About 80Mb down and 20Mb up just as I started the test) > > > > HFSC: P2P in 20% queue and 80/443/8080 in 40% queue with ACKs going to a 20% realtime queue > > > > http://www.dslreports.com/speedtest/622452 > > > > > > I know this is not really your question, but I think the ACKs should go into the same queue as the matching data packets. Think about it that way, if the data is delayed due to congestion it does not make too much sense to tell the sender to send more faster (which essentially is what ACK prioritization does) as that will not really reduce the congestion but rather increase it. > > > There is one caveat though: when ECN is used it might make sense to send out the ACK that will signal the congestion state back to the sender faster=E2=80=A6 So if you prioritize ACKs only select those with an = ECN-Echo flag ;) > > > @bloat : What do you all think about this refined ACK prioritization scheme? > > > > > > Best Regards > > > Sebastian > > > > Here's a very real problem for many users. If you have a highly asymmetrical connection like what many DSL and cable users have, doing even mild uploading can consume enough bandwidth to affect your ability to upload ACKs, which can negatively affect your downloads. > > A regular offender to many is P2P. If you're uploading while downloading on a 30/3 connection, you may not be able to ACK data fast enough. Of course this is in of itself mostly an issue caused by bufferbloat and/or lack of fair queueing. > > I agree, but I think reducing buffer bloat and using fair queueing is superior to ACK prioritization, that=E2=80=99s all. I wholly agree. A separate ACK queue could help in some very specific cases where a trade off is needed, but letting an AQM do its job is probably better because it should "just work". I will do away with my ACK queue once fq_codel comes to PFSense, but until then, I want to make sure my ACKs are not getting stuck behind some burst of traffic. > > > > > I guess the real question is, should ACKs be prioritized in a system that does not exhibit bufferbloat or has fair queuing that allows the ACKs to not feel the affect of other bandwidth being consumed. With no data to back this up, my first guess would be it will not matter. If you have no bufferbloat or you have fair queuing, the ACKs should effectively be sent nearly immediately. > > Actually fq_codel mildly boosts sparse flows, and since ACK typically are sparse they usually do not suffer. But all sparse or starting flows get the same treatment, so nothing ACK specific just something that typically works well. > > > One case where it could make a difference if where ACKs make up a sizable portion of the upload bandwidth if download is saturated. An example would be a hyper-asymmetrical connection like 60/3. > > Okay, I will bite, at one ACK every 2 full MSS send, ACKs take up roughly 2% of the reverse traffic, so at 60 the ACK will cost 60*0.02 =3D 1.2 or 100*1.2/3 =3D 40%, and I agree that is most likely will cause problems for people using smaller priority queues. It is also one reason why I have read recommendations to size priority classes equally (percentage wise) for up- and download, and make sure ACKs get the same priority as the reverse data... > > > In this case, having the ACKs in a separate queue would actually do the opposite, it would put an upper bound on how much bandwidth ACKs could consume unless you gave your ACK queue nearly all of the bandwidth. > > Well, as stated above if your incoming traffic class allows 100% bandwidth use the corresponding outgoing class would better allow the same percentage=E2=80=A6 So i still think not putting ACKs in higher priorities = gives better overall system balance. BUT I have no experience with P2P traffic, so things could be different there. I would hope to be able to tell my P2P apps to mark their packets as background priority, so the data and ACKs would move out of the way of more urgent traffic. I understand that you have actual P2P experience so take my input with a grain of salt, please... > > Best Regards > Sebastian My P2P experience is limited at best. I'm on a dedicated symmetrical connection and my ISP already has anti-bufferbloat stuff implemented. Most of what I hear about using separate ACK queues is entirely because of bufferbloat, you don't want your ACKs stuck behind 1sec of bloat or getting dropped. Until bufferbloat is mostly solved, there probably won't be enough user stories to make a case one way or the other about how modern AQMs will affect ACKs under a wide range of loads, technologies, topologies, and bandwidths. Don't fix what ain't broken and KISS. If you have access to a reliable AQM, don't use an ACK queue unless you have proof that it'll help. > > > > > In the two examples that I could quickly think of, either it made little difference or it did the opposite of what your proposed. Of course it depends on the configuration of the shaping. > > > > Minor side tangent about bloat and ACKs that doesn't really need a discussion > > > > A few months back I was downloading some Linux torrents when I noticed I was getting packetloss. I pretty much never get packetloss. I also noticed that I was receiving a fluctuating 100Mb/s-103Mb/s of ingress on my WAN but only seeing about 80Mb/s of egress on my LAN. I did a packet capture and saw something funny. My WAN was sending out a lot of Dup ACK packets. > > I grabbed a few of the dest IPs that my firewall was sending to and did a trace route. Nice low pings for most of the path, then suddenly about 2-3 hops before the seeder in case their ISP's network, I started to see huge jitter and pings, in the 1sec-3sec range. Cox and Comcast were the two main offenders but they do represent a sizable portion of the population. > > My conclusion was bufferbloat was so bad on their networks that the seeders where not getting ACKs from me in a timely fashion, so they resent the segments assuming they were lost. This issue was so bad that about 20Mb/s of the 100Mb/s was duplicate data segments. I was being DDOS'd by bufferbloat. > > > > P.S. I removed the emails because I was not absolutely sure if they would be scrubbed. --001a1149424ccc43970518567fea Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
>= Hi Benjamin,
>=C2=A0
> On Jun 12, 2015, at 17:33 , Benjamin Cronce <bcronce = at gmail.com> wrote:
>=C2=A0
> > > On = Fri, Jun 12, 2015 at 4:08 AM, Sebastian Moeller wrote:
> > > Hi Benjamin,
>= ; > >=C2=A0
> > > To go off = onto a tangent:
> > >=C2=A0
<= div class=3D"gmail_extra">> > > On Jun 12, 2015, at 06:45 , Benjam= in Cronce wrote:
> > >=C2=A0
=
> > > > [...]
> > > > Under load while doing P2P(About 80Mb down an= d 20Mb up just as I started the test)
> = > > > HFSC: P2P in 20% queue and 80/443/8080 in 40% queue with ACK= s going to a 20% realtime queue
> > &= gt; > http://www.= dslreports.com/speedtest/622452
> &g= t; >=C2=A0
> > > =C2=A0 =C2=A0 = =C2=A0 =C2=A0 I know this is not really your question, but I think the ACKs= should go into the same queue as the matching data packets. Think about it= that way, if the data is delayed due to congestion it does not make too mu= ch sense to tell the sender to send more faster (which essentially is what = ACK prioritization does) as that will not really reduce the congestion but = rather increase it.
> > > =C2=A0 = =C2=A0 =C2=A0 =C2=A0 There is one caveat though: when ECN is used it might = make sense to send out the ACK that will signal the congestion state back t= o the sender faster=E2=80=A6 So if you prioritize ACKs only select those wi= th an ECN-Echo flag ;)
> > > =C2= =A0 =C2=A0 =C2=A0 =C2=A0 @bloat : What do you all think about this refined = ACK prioritization scheme?
> > >= =C2=A0
> > > Best Regards
> > > =C2=A0 =C2=A0 =C2=A0 =C2=A0 Sebastia= n
> >=C2=A0
> > Here's a very real problem for many users. If you have= a highly asymmetrical connection like what many DSL and cable users have, = doing even mild uploading can consume enough bandwidth to affect your abili= ty to upload ACKs, which can negatively affect your downloads.
> > A regular offender to many is P2P. If you'= ;re uploading while downloading on a 30/3 connection, you may not be able t= o ACK data fast enough. Of course this is in of itself mostly an issue caus= ed by bufferbloat and/or lack of fair queueing.
>=C2=A0
> I agree, but I think reducing buffer bloat and= using fair queueing is superior to ACK prioritization, that=E2=80=99s all.=

I who= lly agree. A separate ACK queue could help in some very specific cases wher= e a trade off is needed, but letting an AQM do its job is probably better b= ecause it should "just work".
I w= ill do away with my ACK queue once fq_codel comes to PFSense, but until the= n, I want to make sure my ACKs are not getting stuck behind some burst of t= raffic.

>=C2=A0
> >=C2=A0
> > I guess the real question is, should ACKs be pr= ioritized in a system that does not exhibit bufferbloat or has fair queuing= that allows the ACKs to not feel the affect of other bandwidth being consu= med. With no data to back this up, my first guess would be it will not matt= er. If you have no bufferbloat or you have fair queuing, the ACKs should ef= fectively be sent nearly immediately.
>= =C2=A0
> Actually fq_codel mildly boosts sparse flows, and since= ACK typically are sparse they usually do not suffer. But all sparse or sta= rting flows get the same treatment, so nothing ACK specific just something = that typically works well.=C2=A0
>=C2=A0=
> > One case where it could make a d= ifference if where ACKs make up a sizable portion of the upload bandwidth i= f download is saturated. An example would be a hyper-asymmetrical connectio= n like 60/3.
>=C2=A0
> Okay, = I will bite, at one ACK every 2 full MSS send, ACKs take up roughly 2% of t= he reverse traffic, so at 60 the ACK will cost 60*0.02 =3D 1.2 or 100*1.2/3= =3D 40%, and I agree that is most likely will cause problems for people us= ing smaller priority queues. It is also one reason why I have read recommen= dations to size priority classes equally (percentage wise) for up- and down= load, and make sure ACKs get the same priority as the reverse data...
=
>=C2=A0
> = > In this case, having the ACKs in a separate queue would actually do th= e opposite, it would put an upper bound on how much bandwidth ACKs could co= nsume unless you gave your ACK queue nearly all of the bandwidth.
>=C2=A0
> Well, as stated above if you= r incoming traffic class allows 100% bandwidth use the corresponding outgoi= ng class would better allow the same percentage=E2=80=A6 So i still think n= ot putting ACKs in higher priorities gives better overall system balance. B= UT I have no experience with P2P traffic, so things could be different ther= e. I would hope to be able to tell my P2P apps to mark their packets as bac= kground priority, so the data and ACKs would move out of the way of more ur= gent traffic. I understand that you have actual P2P experience so take my i= nput with a grain of salt, please...
>= =C2=A0
> Best Regards
> Sebas= tian

M= y P2P experience is limited at best. I'm on a dedicated symmetrical con= nection and my ISP already has anti-bufferbloat stuff implemented. Most of = what I hear about using separate ACK queues is entirely because of bufferbl= oat, you don't want your ACKs stuck behind 1sec of bloat or getting dro= pped.
Until bufferbloat is mostly solved, t= here probably won't be enough user stories to make a case one way or th= e other about how modern AQMs will affect ACKs under a wide range of loads,= technologies, topologies, and bandwidths. Don't fix what ain't bro= ken and KISS. If you have access to a reliable AQM, don't use an ACK qu= eue unless you have proof that it'll help.

>=C2=A0
> >=C2=A0
> > In the t= wo examples that I could quickly think of, either it made little difference= or it did the opposite of what your proposed. Of course it depends on the = configuration of the shaping.
> >=C2= =A0
> > Minor side tangent about bloa= t and ACKs that doesn't really need a discussion
> >=C2=A0
> > A few m= onths back I was downloading some Linux torrents when I noticed I was getti= ng packetloss. I pretty much never get packetloss. I also noticed that I wa= s receiving a fluctuating 100Mb/s-103Mb/s of ingress on my WAN but only see= ing about 80Mb/s of egress on my LAN. I did a packet capture and saw someth= ing funny. My WAN was sending out a lot of Dup ACK packets.
> > I grabbed a few of the dest IPs that my firewall= was sending to and did a trace route. Nice low pings for most of the path,= then suddenly about 2-3 hops before the seeder in case their ISP's net= work, I started to see huge jitter and pings, in the 1sec-3sec range. Cox a= nd Comcast were the two main offenders but they do represent a sizable port= ion of the population.
> > My conclus= ion was bufferbloat was so bad on their networks that the seeders where not= getting ACKs from me in a timely fashion, so they resent the segments assu= ming they were lost. This issue was so bad that about 20Mb/s of the 100Mb/s= was duplicate data segments. I was being DDOS'd by bufferbloat.
<= div class=3D"gmail_extra">> >=C2=A0
&= gt; > P.S. I removed the emails because I was not absolutely sure if the= y would be scrubbed.
--001a1149424ccc43970518567fea--