From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-qt1-x842.google.com (mail-qt1-x842.google.com [IPv6:2607:f8b0:4864:20::842]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id AFCFD3BA8E; Tue, 27 Nov 2018 15:51:09 -0500 (EST) Received: by mail-qt1-x842.google.com with SMTP id y20so23406693qtm.13; Tue, 27 Nov 2018 12:51:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=Clo8EKyNZqcGFa1bl984+mRWLIhRXgJsxy9uLERrF/k=; b=rYKcmbRYTipDAUJsiCIUW6phy+eU/nIEu6cAsOLfLyd/AealjD1f7jMqJzFfUHoAJI Gtn53byrH7OY4aXNICCMrN+z+Z9VknX9HLC9Jte+S1kujex+LJ120pl/JUlklmcgPK42 elzX7WMRFImcGCKY0UJm3wHLEk4VcabIrii6QHcE5bkCk98yXBq8Eh00ULvPybC98pQ4 Of5I/+Wph3IrWbPS5N+cx8ZXOHigP+/Hnc3Q2DC7juZF6hVf+dJou15YqrYxz+t4m/rC Sqri0LjhOZEAAd2TYEpgaUVNvXspLuYpXWGFXGyXMoh+EFe+0yo8sfcK8V8UaGlX1R/c kJPw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=Clo8EKyNZqcGFa1bl984+mRWLIhRXgJsxy9uLERrF/k=; b=WU03rN+BFvP55Vthtak/s9LqvBITvcPZO/dk8kRciimf6/HCeNNYQYGHM7UvyHtZPy 0SlbO78O52tq4mfS7BL38J66j97gMxum4eLDKVd9gQK9XPMIruXyHGn70Zs0a+3tZ2DY SfDbaaSpwnQWz5MWfIA+qJeg26h77099b2ZrSsx4gjXL/Fh1v2LkWB6M2RK0bZOEPxGk iw3TjxsUOQOecncfE2pJkuWVWAf/g8qcyLU07lrA2LDEQ0kWv3OTSPY7ReSNUafPrxw6 U2HVz8hw42FLc0q9HBm+xmfqu1iE1zJIS0JqcQuOCsUAVbC+mQYGNYheLkWoRrKv+nCd 84SQ== X-Gm-Message-State: AA+aEWaHo3QeWP+Vxr+43TuMIE+HgILeR2jskqCtFSqtU3SXTKeixCuG PS6EVduiHdDXGChncISU6+5QlTZf5eU1gZymGsk= X-Google-Smtp-Source: AFSGD/UydcXG2f2FE+DCPmQ+i+T2uWEpIJXAerVOhkXTNiMkinaLWpMkFcCr7xKB+zsa5IFGVHxpQoypKeHpZx0pnrU= X-Received: by 2002:ad4:420a:: with SMTP id k10mr32004946qvp.206.1543351869113; Tue, 27 Nov 2018 12:51:09 -0800 (PST) MIME-Version: 1.0 References: <65EAC6C1-4688-46B6-A575-A6C7F2C066C5@heistp.net> In-Reply-To: From: Dave Taht Date: Tue, 27 Nov 2018 12:50:57 -0800 Message-ID: To: Jonathan Morton Cc: Pete Heist , bloat , ecn-sane@lists.bufferbloat.net Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Subject: Re: [Ecn-sane] [Bloat] when does the CoDel part of fq_codel help in the real world? X-BeenThere: ecn-sane@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion of explicit congestion notification's impact on the Internet List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Nov 2018 20:51:09 -0000 On Mon, Nov 26, 2018 at 1:30 PM Jonathan Morton wro= te: > > > On 26 Nov, 2018, at 9:08 pm, Pete Heist wrote: > > > > So I just thought to continue the discussion- when does the CoDel part = of fq_codel actually help in the real world? > > Fundamentally, without Codel the only limits on the congestion window wou= ld be when the sender or receiver hit configured or calculated rwnd and cwn= d limits (the rwnd is visible on the wire and usually chosen to be large en= ough to be a non-factor), or when the queue overflows. Large windows requi= re buffer memory in both sender and receiver, increasing costs on the sende= r in particular (who typically has many flows to manage per machine). You can run devices out of memory more easily with our current ecn implementations. I am seeing folk cut memory per instance to 256k in, for example, the gluon project. we end up dropping (which is better than the device crashing), and in the fq_codel case, *bulk* head dropping. I see a bifurcation on the data when we do this, and I have a one line patch to the fq_codel bulk dropper that appears to make things better when extremely memory constrained like this, but haven't got around to fully evaluating it: https://github.com/dtaht/fq_codel_fast/commit/a524fc2e39dc291199b9b04fb890e= a1548f17641 would rather like more to try the memory limits we are seeing in the field. 32MB (fq_codel default) is waaaay too much. 4MB is way too much even for gbit, I think. 256k? well, given the choice between crashing or not... > > Queue overflow tends to result in burst loss and head-of-line blocking in= the receiver, which is visible to the user as a pause and subsequent jump = in the progress of their download, accompanied by a major fluctuation in th= e estimated time to completion. The lost packets also consume capacity ups= tream of the bottleneck which does not contribute to application throughput= . These effects are independent of whether overflow dropping occurs at the= head or tail of the bottleneck queue, though recovery occurs more quickly = (and fewer packets might be lost) if dropping occurs from the head of the q= ueue. > > From a pure throughput-efficiency standpoint, Codel allows using ECN for = congestion signalling instead of packet loss, potentially eliminating packe= t loss and associated lead-of-line blocking entirely. Even without ECN, th= e actual cwnd is kept near the minimum necessary to satisfy the BDP of the = path, reducing memory requirements and significantly shortening the recover= y time of each loss cycle, to the point where the end-user may not notice t= hat delivery is not perfectly smooth, and implementing accurate completion = time estimators is considerably simplified. I wish we had fractional cwnd below 1 and/or that pacing did not rely on cwnd at all. too many flows at any rate you choose, can end up marking 100% of packets, still not run you out of memory, and cause delay. > > An important use-case is where two sequential bottlenecks exist on the pa= th, the upstream one being only slightly higher capacity but lacking any qu= eue management at all. This is presently common in cases where home CPE im= plements inbound shaping on a generic ISP last-mile link. In that case, wi= thout Codel running on the second bottleneck, traffic would collect in the = first bottleneck's queue as well, greatly reducing the beneficial effects o= f FQ implemented on the second bottleneck. In this topology, the overall e= ffect is inter-flow as well as intra-flow. > > The combination of Codel with FQ is done in such a way that a separate in= stance of Codel is implemented for each flow. This means that congestion s= ignals are only sent to flows that require them, and non-saturating flows a= re unmolested. This makes the combination synergistic, where each componen= t offers an improvement to the behaviour of the other. also a good bullet point somewhere! > > - Jonathan Morton > > _______________________________________________ > Bloat mailing list > Bloat@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/bloat --=20 Dave T=C3=A4ht CTO, TekLibre, LLC http://www.teklibre.com Tel: 1-831-205-9740