From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-qk1-x72d.google.com (mail-qk1-x72d.google.com [IPv6:2607:f8b0:4864:20::72d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 752543BA8E for ; Tue, 27 Nov 2018 17:19:27 -0500 (EST) Received: by mail-qk1-x72d.google.com with SMTP id o125so15679598qkf.3 for ; Tue, 27 Nov 2018 14:19:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=Bt0MGgI5ETqdxJx/Vw6H8YOxlNs1G3zfjWm5PHi3XpU=; b=p+d4GH3McXDj06ErM0xC5j5mHivXHJnzJbs6BJYc+E2uFZgTiz5NnRZM2rdAKeMoxW t2jLsUXj5dz/X3zxqzdWf3pDsswUfRCB5mPpW8+AXPWu8XbjR5AswFu55f0s4H1MGua3 rjUPKN1NEx7d90TNEPhND5RWveEjGLUoNz3P9CQExZ6uj99OSJBN+jonAdl2I54JKWJA UDg1Yzyubqn9EhRFosCStTpTHzmAvqR4iuIUcaAAv4nsVC8kxM063PrEIqG1fNaG0cyk /zeFVmVGiikBTv02+HFaodKvtHIe1R4jG6INh5ppEyt198k6snTnjpfcWfEDtjWhTXKs 96Eg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=Bt0MGgI5ETqdxJx/Vw6H8YOxlNs1G3zfjWm5PHi3XpU=; b=lDJxbkYlpLYYxSWBys+YtRBBdUsDH60PRBBHx8Xlqr/t49UXnSpvMkn4ZlN0xxV9d1 0bsFLRZrPOnx+gnmSg8loxYxcileGJgXGJcVi3t9pHlxlmYS52jGnP9f9ERa0Apa84FP eAjNIG32gHa89iCNosMd6HJulJdA3Xp0MFaDlcqCrvJ+3bPIBrY2hdBBxpdZApHRoJfV Bpp3cMqXzbgFVQD3rYVARCwY9uUA51SscbyOkZMqkP1lU5fD4DYhJ+4iFK2HyKrbtPvs AglcF/rcMD9O2p7WyxZ2Tp+BMbxyMnbNvALISSpuJh+Ni0Q9qQrpsRnb8TniwHcfWlcL PpEA== X-Gm-Message-State: AA+aEWbjCe/Pcsbv+71cnAzTa5G/owG+7PZ9Juk+y8Dpe5hawyn1Mzvo adNWplTkT01Uzo26i5J4R1OxwyPn+j4Hr11xGw8= X-Google-Smtp-Source: AFSGD/WsP44LZFJ80UKQdhhzFwBaHWZvAbNRysfnyyyRDol+7zFn6L6XDGKMrK/1uG3a894U9nAoclbL3UhJCvFlvx8= X-Received: by 2002:a37:d994:: with SMTP id q20mr31473268qkl.116.1543357166775; Tue, 27 Nov 2018 14:19:26 -0800 (PST) MIME-Version: 1.0 References: <65EAC6C1-4688-46B6-A575-A6C7F2C066C5@heistp.net> In-Reply-To: From: Luca Muscariello Date: Tue, 27 Nov 2018 23:19:15 +0100 Message-ID: To: Dave Taht Cc: Jonathan Morton , bloat Content-Type: multipart/alternative; boundary="000000000000aff32b057bacd7b7" Subject: Re: [Bloat] when does the CoDel part of fq_codel help in the real world? X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Nov 2018 22:19:27 -0000 --000000000000aff32b057bacd7b7 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable I suggest re-reading this https://queue.acm.org/detail.cfm?id=3D3022184 On Tue 27 Nov 2018 at 21:58, Dave Taht wrote: > OK, wow, this conversation got long. and I'm still 20 messages behind. > > Two points, and I'm going to go back to work, and maybe I'll try to > summarize a table > of the competing viewpoints, as there's far more than BDP of > discussion here, and what > we need is sqrt(bdp) to deal with all the different conversational flows. > :) > > On Tue, Nov 27, 2018 at 1:24 AM Luca Muscariello > wrote: > > > > I think that this is a very good comment to the discussion at the > defense about the comparison between > > SFQ with longest queue drop and FQ_Codel. > > > > A congestion controlled protocol such as TCP or others, including QUIC, > LEDBAT and so on > > need at least the BDP in the transmission queue to get full link > efficiency, i.e. the queue never empties out. > > no, I think it needs a BDP in flight. > > I think some of the confusion here is that your TCP stack needs to > keep around a BDP in order to deal with > retransmits, but that lives in another set of buffers entirely. > > > This gives rule of thumbs to size buffers which is also very practical > and thanks to flow isolation becomes very accurate. > > > > Which is: > > > > 1) find a way to keep the number of backlogged flows at a reasonable > value. > > This largely depends on the minimum fair rate an application may need i= n > the long term. > > We discussed a little bit of available mechanisms to achieve that in th= e > literature. > > > > 2) fix the largest RTT you want to serve at full utilization and size > the buffer using BDP * N_backlogged. > > Or the other way round: check how much memory you can use > > in the router/line card/device and for a fixed N, compute the largest > RTT you can serve at full utilization. > > My own take on the whole BDP argument is that *so long as the flows in > that BDP are thoroughly mixed* you win. > > > > > 3) there is still some memory to dimension for sparse flows in addition > to that, but this is not based on BDP. > > It is just enough to compute the total utilization of sparse flows and > use the same simple model Toke has used > > to compute the (de)prioritization probability. > > > > This procedure would allow to size FQ_codel but also SFQ. > > It would be interesting to compare the two under this buffer sizing. > > It would also be interesting to compare another mechanism that we have > mentioned during the defense > > which is AFD + a sparse flow queue. Which is, BTW, already available in > Cisco nexus switches for data centres. > > > > I think that the the codel part would still provide the ECN feature, > that all the others cannot have. > > However the others, the last one especially can be implemented in > silicon with reasonable cost. > > > > > > > > > > > > On Mon 26 Nov 2018 at 22:30, Jonathan Morton > wrote: > >> > >> > On 26 Nov, 2018, at 9:08 pm, Pete Heist wrote: > >> > > >> > So I just thought to continue the discussion- when does the CoDel > part of fq_codel actually help in the real world? > >> > >> Fundamentally, without Codel the only limits on the congestion window > would be when the sender or receiver hit configured or calculated rwnd an= d > cwnd limits (the rwnd is visible on the wire and usually chosen to be lar= ge > enough to be a non-factor), or when the queue overflows. Large windows > require buffer memory in both sender and receiver, increasing costs on th= e > sender in particular (who typically has many flows to manage per machine)= . > >> > >> Queue overflow tends to result in burst loss and head-of-line blocking > in the receiver, which is visible to the user as a pause and subsequent > jump in the progress of their download, accompanied by a major fluctuatio= n > in the estimated time to completion. The lost packets also consume > capacity upstream of the bottleneck which does not contribute to > application throughput. These effects are independent of whether overflo= w > dropping occurs at the head or tail of the bottleneck queue, though > recovery occurs more quickly (and fewer packets might be lost) if droppin= g > occurs from the head of the queue. > >> > >> From a pure throughput-efficiency standpoint, Codel allows using ECN > for congestion signalling instead of packet loss, potentially eliminating > packet loss and associated lead-of-line blocking entirely. Even without > ECN, the actual cwnd is kept near the minimum necessary to satisfy the BD= P > of the path, reducing memory requirements and significantly shortening th= e > recovery time of each loss cycle, to the point where the end-user may not > notice that delivery is not perfectly smooth, and implementing accurate > completion time estimators is considerably simplified. > >> > >> An important use-case is where two sequential bottlenecks exist on the > path, the upstream one being only slightly higher capacity but lacking an= y > queue management at all. This is presently common in cases where home CP= E > implements inbound shaping on a generic ISP last-mile link. In that case= , > without Codel running on the second bottleneck, traffic would collect in > the first bottleneck's queue as well, greatly reducing the beneficial > effects of FQ implemented on the second bottleneck. In this topology, th= e > overall effect is inter-flow as well as intra-flow. > >> > >> The combination of Codel with FQ is done in such a way that a separate > instance of Codel is implemented for each flow. This means that congesti= on > signals are only sent to flows that require them, and non-saturating flow= s > are unmolested. This makes the combination synergistic, where each > component offers an improvement to the behaviour of the other. > >> > >> - Jonathan Morton > >> > >> _______________________________________________ > >> Bloat mailing list > >> Bloat@lists.bufferbloat.net > >> https://lists.bufferbloat.net/listinfo/bloat > > > > _______________________________________________ > > Bloat mailing list > > Bloat@lists.bufferbloat.net > > https://lists.bufferbloat.net/listinfo/bloat > > > > -- > > Dave T=C3=A4ht > CTO, TekLibre, LLC > http://www.teklibre.com > Tel: 1-831-205-9740 > --000000000000aff32b057bacd7b7 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
I suggest re-reading this=C2=A0


On Tue= 27 Nov 2018 at 21:58, Dave Taht <dave.taht@gmail.com> wrote:
OK, wow, this conversation got long. and I'm still 20 messages behind.=

Two points, and I'm going to go back to work, and maybe I'll try to=
summarize a table
of the competing viewpoints, as there's far more than BDP of
discussion here, and what
we need is sqrt(bdp) to deal with all the different conversational flows. := )

On Tue, Nov 27, 2018 at 1:24 AM Luca Muscariello
<luca.mu= scariello@gmail.com> wrote:
>
> I think that this is a very good comment to the discussion at the defe= nse about the comparison between
> SFQ with longest queue drop and FQ_Codel.
>
> A congestion controlled protocol such as TCP or others, including QUIC= , LEDBAT and so on
> need at least the BDP in the transmission queue to get full link effic= iency, i.e. the queue never empties out.

no, I think it needs a BDP in flight.

I think some of the confusion here is that your TCP stack needs to
keep around a BDP in order to deal with
retransmits, but that lives in another set of buffers entirely.

> This gives rule of thumbs to size buffers which is also very practical= and thanks to flow isolation becomes very accurate.
>
> Which is:
>
> 1) find a way to keep the number of backlogged flows at a reasonable v= alue.
> This largely depends on the minimum fair rate an application may need = in the long term.
> We discussed a little bit of available mechanisms to achieve that in t= he literature.
>
> 2) fix the largest RTT you want to serve at full utilization and size = the buffer using BDP * N_backlogged.
> Or the other way round: check how much memory you can use
> in the router/line card/device and for a fixed N, compute the largest = RTT you can serve at full utilization.

My own take on the whole BDP argument is that *so long as the flows in
that BDP are thoroughly mixed* you win.

>
> 3) there is still some memory to dimension for sparse flows in additio= n to that, but this is not based on BDP.
> It is just enough to compute the total utilization of sparse flows and= use the same simple model Toke has used
> to compute the (de)prioritization probability.
>
> This procedure would allow to size FQ_codel but also SFQ.
> It would be interesting to compare the two under this buffer sizing. > It would also be interesting to compare another mechanism that we have= mentioned during the defense
> which is AFD + a sparse flow queue. Which is, BTW, already available i= n Cisco nexus switches for data centres.
>
> I think that the the codel part would still provide the ECN feature, t= hat all the others cannot have.
> However the others, the last one especially can be implemented in sili= con with reasonable cost.
>
>
>
>
>
> On Mon 26 Nov 2018 at 22:30, Jonathan Morton <chromatix99@gmail.com> wrote:<= br> >>
>> > On 26 Nov, 2018, at 9:08 pm, Pete Heist <pete@heistp.net> wrote:
>> >
>> > So I just thought to continue the discussion- when does the C= oDel part of fq_codel actually help in the real world?
>>
>> Fundamentally, without Codel the only limits on the congestion win= dow would be when the sender or receiver hit configured or calculated rwnd = and cwnd limits (the rwnd is visible on the wire and usually chosen to be l= arge enough to be a non-factor), or when the queue overflows.=C2=A0 Large w= indows require buffer memory in both sender and receiver, increasing costs = on the sender in particular (who typically has many flows to manage per mac= hine).
>>
>> Queue overflow tends to result in burst loss and head-of-line bloc= king in the receiver, which is visible to the user as a pause and subsequen= t jump in the progress of their download, accompanied by a major fluctuatio= n in the estimated time to completion.=C2=A0 The lost packets also consume = capacity upstream of the bottleneck which does not contribute to applicatio= n throughput.=C2=A0 These effects are independent of whether overflow dropp= ing occurs at the head or tail of the bottleneck queue, though recovery occ= urs more quickly (and fewer packets might be lost) if dropping occurs from = the head of the queue.
>>
>> From a pure throughput-efficiency standpoint, Codel allows using E= CN for congestion signalling instead of packet loss, potentially eliminatin= g packet loss and associated lead-of-line blocking entirely.=C2=A0 Even wit= hout ECN, the actual cwnd is kept near the minimum necessary to satisfy the= BDP of the path, reducing memory requirements and significantly shortening= the recovery time of each loss cycle, to the point where the end-user may = not notice that delivery is not perfectly smooth, and implementing accurate= completion time estimators is considerably simplified.
>>
>> An important use-case is where two sequential bottlenecks exist on= the path, the upstream one being only slightly higher capacity but lacking= any queue management at all.=C2=A0 This is presently common in cases where= home CPE implements inbound shaping on a generic ISP last-mile link.=C2=A0= In that case, without Codel running on the second bottleneck, traffic woul= d collect in the first bottleneck's queue as well, greatly reducing the= beneficial effects of FQ implemented on the second bottleneck.=C2=A0 In th= is topology, the overall effect is inter-flow as well as intra-flow.
>>
>> The combination of Codel with FQ is done in such a way that a sepa= rate instance of Codel is implemented for each flow.=C2=A0 This means that = congestion signals are only sent to flows that require them, and non-satura= ting flows are unmolested.=C2=A0 This makes the combination synergistic, wh= ere each component offers an improvement to the behaviour of the other.
>>
>>=C2=A0 - Jonathan Morton
>>
>> _______________________________________________
>> Bloat mailing list
>> B= loat@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/bloat
>
> _______________________________________________
> Bloat mailing list
> Bloat= @lists.bufferbloat.net
> https://lists.bufferbloat.net/listinfo/bloat


--

Dave T=C3=A4ht
CTO, TekLibre, LLC
ht= tp://www.teklibre.com
Tel: 1-831-205-9740
--000000000000aff32b057bacd7b7--