From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-yh0-x241.google.com (mail-yh0-x241.google.com [IPv6:2607:f8b0:4002:c01::241]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by huchra.bufferbloat.net (Postfix) with ESMTPS id 28B9921F3E2 for ; Fri, 12 Jun 2015 14:19:08 -0700 (PDT) Received: by yhzz6 with SMTP id z6so449074yhz.3 for ; Fri, 12 Jun 2015 14:19:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=3adPNjdk+q6rz/e36Y2t+NKTvFpEXEdJf+8e12xp4hk=; b=RX3ZQhPhKF7t7CKeGonks+F4xLu0RUXkWnzqufFpJMNfpQtpYxG6NdfDiQge/n7ix+ 1EkTIZYCqLAqLMZkdrTW0ns37c5V7ycmjn/oIK/YsMl5UmRpC+0wgzzwYIpL9nNDI0DX RLoYBsFw3z17i37Kyt6jl2PLxuWQGMwOmRtPDkbPZi9kk8b5BKco0Px08LFcRNJa+o+u ieQ2Mv5/XCK6DOueSyn/rqAq7/g6SLKuktWwosjITFJEvX+5Ds17D4oc1oU8WryOdbC3 9YRjVdiAaFFbkaWwcLe7AHKiJbx62F6XL5FnUcv4HXnLWnucYeFEivhw6glDWoL/JUoF 1aQg== MIME-Version: 1.0 X-Received: by 10.13.203.193 with SMTP id n184mr20855725ywd.10.1434143947769; Fri, 12 Jun 2015 14:19:07 -0700 (PDT) Received: by 10.129.148.194 with HTTP; Fri, 12 Jun 2015 14:19:07 -0700 (PDT) In-Reply-To: <628F8092-841C-42DA-861B-C60C36457B04@gmx.de> References: <6AD8E150-5751-43AC-8F6C-8175C1E92DE1@gmx.de> <628F8092-841C-42DA-861B-C60C36457B04@gmx.de> Date: Fri, 12 Jun 2015 16:19:07 -0500 Message-ID: From: Benjamin Cronce To: Sebastian Moeller Content-Type: multipart/alternative; boundary=001a11482858904e2e051858a8ea Cc: Jonathan Morton , Alex Elsayed , bloat Subject: Re: [Bloat] Bloat done correctly? X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Jun 2015 21:19:37 -0000 --001a11482858904e2e051858a8ea Content-Type: text/plain; charset=UTF-8 > Hi Jonathan, > > On June 12, 2015 9:14:02 PM GMT+02:00, Jonathan Morton wrote: > >We have a test in Flent which tries to exercise this case: 50 flows in one > >direction and 1 in the other, all TCP. Where the 50 flows are on the narrow > >side of an asymmetric link, it is possible to see just what happens when > >there isn't enough bandwidth for the acks of the single opposing flow. > > > >What I see is that acks behave like an unresponsive flow in themselves, but > >one that is reasonably tolerant to loss (more so than to delay). On a > >standard AQM, the many flows end up yielding to the acks; on a > >flow-isolating AQM, the acks are restricted to a fair (1/51) share, but > >enough of them are dropped to (eventually) let the opposing flow get most > >of the available bandwidth on its side. But on an FQ without AQM, acks > >don't get dropped so they get delayed instead, and the opposing flow will > >be ack clocked to a limited bandwidth until the ack queue overflows. > > > >Cake ends up causing odd behaviour this way. I have a suspicion about why > >one of the weirder effects shows up - it has to get so aggressive about > >dropping acks that the count variable for that queue wraps around. > >Implementing saturating arithmetic there might help. > > > >There is a proposed TCP extension for ack congestion control, which allows > >the ack ratio to be varied in response to ack losses. This would be a > >cleaner way to achieve the same effect, and would allow enabling ECN on the > >acks, but it's highly experimental. > > This is reducing the ACK-rate to make losses less likely, but at the same time it makes a single loss more costly, so whether this is a win depends on whether the sparser ACK flow has a much higher probability to pass trough the congested link. I wonder what percentage of an ACK flow can be dropped without slowing the sender? > > > > >- Jonathan Morton I also question the general usefulness of sparser ACK flows just to accommodate hyper-asymmetrical connections. The main causes of these issues are old DSL techs or DOCSIS rollouts that haven't fully switched to DOCSIS3. Many cable companies claim to be DOCSIS3 because their downstream is 3.0, but their upstream is 2.0 because 3.0 requires old line-filters to be replaced with newer ones that open up some low frequency channels. Once these channels get opened up, assuming fiber doesn't get their first, there will be a lot more available upstream bandwidth, assuming the ISPs provision it. Modern OSs already support naggle to be toggled on/off for a given TCP flow. Maybe a change to the algorithm to detect TCP RTTs above a threshhold or certain patterns in lost ACKs to trigger increasing the number of packets naggle coalesces for ACKs. My understanding of naggle is it take two parameters, a window and a max number. I think the default window is something like 100ms and the default max coalesced ACKs are two. But you can modify either value. So technically ACK rates can already be modified, it's just not done dynamically, but the feature already exists. Instead of making further changes to TCP, educate people on how to change their TCP settings? I could see this making strange issues with really low RTTs where the RTT is lower than the naggle window making for a small receive window. Because TCP implementations have a minimum of 2 segments, it matches with naggle's default to combine two ACKs. If you suddenly decided to combine 3 segments, two segments get sent, the other side receives the segments but does not ACK them because it's waiting for a 3rd. The other side does not send any more segments because it's waiting for an ACK. You suddenly get these strange pulses based on the naggle window. Because naggle's combine matches perfectly with the minimum outstanding segments, this corner case does not exist. But I'll leave this to people more knowledgeable than me. Just thinking out loud here. --001a11482858904e2e051858a8ea Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
>= Hi Jonathan,
>=C2=A0
> On June 12, 2015 9:14:02 PM GMT+02:00, Jonathan Morton &= lt;chromatix99 at gmail.com> wrote:
> >We have a test in Flent which tries t= o exercise this case: 50 flows in one
> = >direction and 1 in the other, all TCP. Where the 50 flows are on the na= rrow
> >side of an asymmetric link, i= t is possible to see just what happens when
> >there isn't enough bandwidth for the acks of the single oppos= ing flow.
> >
> >What I see is that acks behave like an unresponsive flow = in themselves, but
> >one that is rea= sonably tolerant to loss (more so than to delay). On a
> >standard AQM, the many flows end up yielding to the ac= ks; on a
> >flow-isolating AQM, the a= cks are restricted to a fair (1/51) share, but
> >enough of them are dropped to (eventually) let the opposing fl= ow get most
> >of the available bandw= idth on its side. But on an FQ without AQM, acks
> >don't get dropped so they get delayed instead, and the o= pposing flow will
> >be ack clocked t= o a limited bandwidth until the ack queue overflows.
> >
> >Cake ends up c= ausing odd behaviour this way. I have a suspicion about why
> >one of the weirder effects shows up - it has to g= et so aggressive about
> >dropping ac= ks that the count variable for that queue wraps around.
> >Implementing saturating arithmetic there might help.<= /div>
> >
&= gt; >There is a proposed TCP extension for ack congestion control, which= allows
> >the ack ratio to be varied= in response to ack losses. This would be a
> >cleaner way to achieve the same effect, and would allow enabling = ECN on the
> >acks, but it's high= ly experimental.
>=C2=A0
> =C2=A0 =C2=A0 =C2=A0 =C2=A0This is reducing the ACK-r= ate to make losses less likely, but at the same time it makes a single loss= more costly, so whether this is a win depends on whether the sparser ACK f= low has a much higher probability to pass trough the congested link. I wond= er what percentage of an ACK flow can be dropped without slowing the sender= ?
>=C2=A0
> >
> >- Jonathan Morton

I also que= stion the general usefulness of sparser ACK flows just to accommodate hyper= -asymmetrical connections. The main causes of these issues are old DSL tech= s or DOCSIS rollouts that haven't fully switched to DOCSIS3. Many cable= companies claim to be DOCSIS3 because their downstream is 3.0, but their u= pstream is 2.0 because 3.0 requires old line-filters to be replaced with ne= wer ones that open up some low frequency channels. Once these channels get = opened up, assuming fiber doesn't get their first, there will be a lot = more available upstream bandwidth, assuming the ISPs provision it.

Modern OSs alr= eady support naggle to be toggled on/off for a given TCP flow. Maybe a chan= ge to the algorithm to detect TCP RTTs above a threshhold or certain patter= ns in lost ACKs to trigger increasing the number of packets naggle coalesce= s for ACKs. My understanding of naggle is it take two parameters, a window = and a max number. I think the default window is something like 100ms and th= e default max coalesced ACKs are two. But you can modify either value. So t= echnically ACK rates can already be modified, it's just not done dynami= cally, but the feature already exists. Instead of making further changes to= TCP, educate people on how to change their TCP settings?

I could see this maki= ng strange issues with really low RTTs where the RTT is lower than the nagg= le window making for a small receive window. Because TCP implementations ha= ve a minimum of 2 segments, it matches with naggle's default to combine= two ACKs. If you suddenly decided to combine 3 segments, two segments get = sent, the other side receives the segments but does not ACK them because it= 's waiting for a 3rd. The other side does not send any more segments be= cause it's waiting for an ACK. You suddenly get these strange pulses ba= sed on the naggle window.

Because naggle's combine matches perfectly with the= minimum outstanding segments, this corner case does not exist. But I'l= l leave this to people more knowledgeable than me. Just thinking out loud h= ere.
--001a11482858904e2e051858a8ea--