From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-lf0-x22c.google.com (mail-lf0-x22c.google.com [IPv6:2a00:1450:4010:c07::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 30ABF3BA8E for ; Wed, 18 Apr 2018 11:03:31 -0400 (EDT) Received: by mail-lf0-x22c.google.com with SMTP id r7-v6so3162157lfr.1 for ; Wed, 18 Apr 2018 08:03:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=CISPZ5k2E2bUBNSglZiUfRBGayWSBsLsDYv45SvJRsY=; b=ttEcA9er8NA8H8uPKDZ0kO/oflGVQvk4Hvu4fScZzCoL6Mg8wiR1xn2PsvWnYKOd+z e9W5RMRGmwytkZIeRxBnHMsVdmkFGNNmLFBPVwTmGErl6+bhn+pocnE+dex/EfZbw/z0 2Q5deVaW0tre6prp0rgdjrtSHFcihjQzFUl0WOpVYT3M13pSfqx7tKF/2cEp394lfPZP uGtcIQmlASUz0Yh+L9Q9QAJ78T94XkCBGlcUriRUY+lv7S+uTKNu6ODL7ZHZVhoEekOu bU3XifMTIBX3ZpPeNUfPZi6vRuEvIufgYaYS0Q17rD7qzBfIYg36UGC+DxL3pwXVr+X9 KuNw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=CISPZ5k2E2bUBNSglZiUfRBGayWSBsLsDYv45SvJRsY=; b=mo5Rq8OpNXxiPmZoPlW+T7+mQQ7ATzOCa9QyhPstfDcHCGUS9c3rzxE3Ene0es8vVI kXFjRhhVDTh2pmclAKDVC9839PhKyFhZFCDa9O5cyM0gZbfFyVj9Fldk1Fovw4lUYn/p 3Dp6m05D7FtvvMfOYM+aoPJjMPCN6UNgIfTT7F2G5ioS0PvXX5qtbf1oWzjemU3NyWnk UjUz6vg8hjmOv/MXyN/5LYEiNAt9Ga0O2+7pTTb1un4mj5+HWjhIO4D1Qm6lhzEDh7/Q cLyWnNDa/QfseGBU8n7hgLvYNZv9FaB+8zYIwOv6XXhRTlgdMuIHR962s14JqHcbFiwX Vqbw== X-Gm-Message-State: ALQs6tBNIvSmkDkRqsFYj+GJzkQFVvPrX5XjHLL/51H3UcBBm2Y4baQV AAHkTmTHW//ejb8Bi+8oHfo= X-Google-Smtp-Source: AIpwx4/exI4cc4jskNBk6OJ+LALLc6sVSR3jY9YTm6u5aZ7w8T5bt70R476V7vKkbTVJ3OXw2TnOBQ== X-Received: by 2002:a19:fa7:: with SMTP id 39-v6mr1637275lfp.138.1524063809940; Wed, 18 Apr 2018 08:03:29 -0700 (PDT) Received: from [192.168.239.216] (83-245-234-255-nat-p.elisa-mobile.fi. [83.245.234.255]) by smtp.gmail.com with ESMTPSA id k185-v6sm304813lfe.96.2018.04.18.08.03.28 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 18 Apr 2018 08:03:29 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 11.3 \(3445.6.18\)) From: Jonathan Morton In-Reply-To: <87r2nc1taq.fsf@toke.dk> Date: Wed, 18 Apr 2018 18:03:26 +0300 Cc: cake@lists.bufferbloat.net Content-Transfer-Encoding: quoted-printable Message-Id: References: <87vacq419h.fsf@toke.dk> <874lk9533l.fsf@toke.dk> <87604o3get.fsf@toke.dk> <578552B2-5127-451A-AFE8-93AE9BB07368@gmail.com> <87r2nc1taq.fsf@toke.dk> To: =?utf-8?Q?Toke_H=C3=B8iland-J=C3=B8rgensen?= X-Mailer: Apple Mail (2.3445.6.18) Subject: Re: [Cake] A few puzzling Cake results X-BeenThere: cake@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Cake - FQ_codel the next generation List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Apr 2018 15:03:31 -0000 >>> So if there is one active bulk flow, we allow each flow to queue = four >>> packets. But if there are ten active bulk flows, we allow *each* = flow to >>> queue *40* packets. >>=20 >> No - because the drain rate per flow scales inversely with the number >> of flows, we have to wait for 40 MTUs' serialisation delay to get 4 >> packets out of *each* flow. >=20 > Ah right, yes. Except it's not 40 MTUs it's 40 quantums (as each flow > will only dequeue a packet each MTU/quantum rounds of the scheduler).=20= The maximum quantum in Cake is equal to the MTU, and obviously you can't = increase the drain rate by decreasing the quantum below the packet size. >> Without that, we can end up with very high drop rates which, in >> ingress mode, don't actually improve congestion on the bottleneck = link >> because TCP can't reduce its window below 4 MTUs, and it's having to >> retransmit all the lost packets as well. That loses us a lot of >> goodput for no good reason. >=20 > I can sorta, maybe, see the point of not dropping packets that won't > cause the flow to decrease its rate *in ingress mode*. But this is = also > enabled in egress mode, where it doesn't make sense. I couldn't think of a good reason to switch it off in egress mode. That = would improve a metric that few people care about or can even measure, = while severely increasing packet loss and retransmissions in some = situations, which is something that people *do* care about and measure. > Also, the minimum TCP window is two packets including those that are = in > flight but not yet queued; so allowing four packets at the bottleneck = is > way excessive. You can only hold the effective congestion window in NewReno down to 2 = packets if you have a 33% AQM signalling rate (dropping one packet per = RTT), which is hellaciously high if the hosts aren't using ECN. If they = *are* using ECN, then goodput in ingress mode doesn't depend inversely = on signalling rate anyway, so it doesn't matter. At 4 packets, the = required signalling rate is still pretty high (1 packet per 3 RTTs, if = it really does go down to 2 MTUs meanwhile) but a lot more manageable - = in particular, it's comfortably within the margin required by ingress = mode - and gets a lot more goodput through. We did actually measure the effect this had in a low-inherent-latency, = low-bandwidth environment. Goodput went up significantly, and peak = inter-flow latency went *down* due to upstream queuing effects. >> So I do accept the increase in intra-flow latency when the flow count >> grows beyond the link's capacity to cope. >=20 > TCP will always increase its bandwidth above the link's capacity to > cope. That's what TCP does. >=20 >> It helps us keep the inter-flow induced latency low >=20 > What does this change have to do with inter-flow latency? >=20 >> while maintaining bulk goodput, which is more important. >=20 > No, it isn't! Accepting a factor of four increase in latency to gain a > few percents' goodput in an edge case is how we got into this whole > bufferbloat mess in the first place... Perhaps a poor choice of wording; I consider *inter-flow latency* to be = the most important factor. But users also consider goodput relative to = link capacity to be important, especially on slow links. Intra-flow = latency, by contrast, is practically invisible except for traffic types = that are usually sparse. As I noted during the thread Kevin linked, Dave originally asserted that = the AQM target should *not* depend on the flow count, but the total = number of packets in the queue should be held constant. I found that = assertion had to be challenged once cases emerged where it was clearly = detrimental. So now I assert the opposite: that the queue must be = capable of accepting a minimum number of packets *per flow*, and not = just transiently, if the inherent latency is not greater than what = corresponds to the optimal BDP for TCP. This tweak has zero theoretical effect on inter-flow latency (which is = guaranteed by the DRR++ scheme, not the AQM), but can improve goodput = and sender load at the expense of intra-flow latency. The practical = effect on inter-flow latency can actually be positive in some scenarios. Feel free to measure. Just be aware of what this is designed to handle. And obviously I need to write about this in the paper... - Jonathan Morton