From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-lf0-x233.google.com (mail-lf0-x233.google.com [IPv6:2a00:1450:4010:c07::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 6C6E13BA8E for ; Tue, 1 May 2018 14:32:02 -0400 (EDT) Received: by mail-lf0-x233.google.com with SMTP id z130-v6so17381139lff.5 for ; Tue, 01 May 2018 11:32:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=yWIFrpE2o1rAHKYucN6k1m/IBP1FERyBhakz00Dierw=; b=NowY8w6qM2Kmio6+TORs+Yfuj9UKwQgaaKNmqiTgIQHznVd6/blvV7ZgXocQLz5NDH tJ8z7/Z2gu1JbpcWW7jdwSnKn6AMzY9YoF4b7niLcuMMM7+ya428TbcMuZg4Zfznrd2d ZumVfZkULHZq9JmAzzqR7RcGovpKWCtqhziogXbCfrtLLtGDzhlcFXdAb4P8Yzqqm5FR VOvwasJIbl4+Cg2Lhv7eNKDs5R8b+svYja1yU75S2inCRXA5LhAjpMLfIhq5SRTFqJFo SdK8TAuAnmP7S2jLw75UYEVaKkBBa4J4lwlwU1wM+dycAsNQVjiFNXtN40/zZb22mt+S bHcg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=yWIFrpE2o1rAHKYucN6k1m/IBP1FERyBhakz00Dierw=; b=fZB86hMLBD4H4iQxBj0lAW6fy8GPa7I/xdDCEEBdyo+0f/1O5ZJ1EAgtePxvhefYRh 7hB6etn150FMBvNmhjp08wFjAwyLAtqZjl7JDS3dA8dyG1MV+id2a3aGVcGCto835kfE iTHV9Eyx23XuVKCuCIi7tdowbhmQ7QFRz9iJr5dVcmXBQA0sKK5pEIGP/MPl4ICzFZFQ +IMaNmoXSQ/Rb0ouTzixYwZIT91IU946gJGHxJkm7/+ux4nBhWC4hP/8OU7yfyGujgAw sYVWlJha03nrFCZsocM0VjuLWZFtNExi6p0sNL72cEEUw+I3Kwuh8R8Bi3PFc1A6ue+L WCmg== X-Gm-Message-State: ALQs6tDr1/TSXU9Ce4dDRQbcoWDdg55Mvy0rYVXyMmbkvr/1VCfJq/xP azewxispAnW7vt+vaWa06SI= X-Google-Smtp-Source: AB8JxZpfYEmE39CyNMut5X31ENakPE/OgM1SVW0W6NamYDOvPhXtSo6ww7gl4Hu2lPljh9PfzX83DA== X-Received: by 2002:a19:d763:: with SMTP id o96-v6mr9782148lfg.89.1525199521187; Tue, 01 May 2018 11:32:01 -0700 (PDT) Received: from [192.168.239.216] (83-245-234-255-nat-p.elisa-mobile.fi. [83.245.234.255]) by smtp.gmail.com with ESMTPSA id h5-v6sm1114148ljb.18.2018.05.01.11.31.59 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 01 May 2018 11:32:00 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 11.3 \(3445.6.18\)) From: Jonathan Morton In-Reply-To: Date: Tue, 1 May 2018 21:31:58 +0300 Cc: Dave Taht , Cong Wang , Cake List , Linux Kernel Network Developers Content-Transfer-Encoding: quoted-printable Message-Id: <5C507E79-DD05-400A-B4A9-364EEEA12C08@gmail.com> References: <20180429213439.7389-1-toke@toke.dk> To: Eric Dumazet X-Mailer: Apple Mail (2.3445.6.18) Subject: Re: [Cake] [PATCH net-next v6] Add Common Applications Kept Enhanced (cake) qdisc X-BeenThere: cake@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Cake - FQ_codel the next generation List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 May 2018 18:32:02 -0000 > On 1 May, 2018, at 7:06 pm, Eric Dumazet = wrote: >=20 > You have not provided any numbers to show how useful it is to maintain = this > code (probably still broken btw, considering it is changing some skb = attributes). A simple calculation shows what the maximum tolerable asymmetry for a = conventional TCP connection is. If the MTU is 1500, a pure ack is 84 = bytes, and an ack is sent for every 3 packets received, bandwidth = asymmetry exceeding about 50:1 will inevitably result in the inability = to fully utilise downstream bandwidth. This ratio drops to 34:1 if acks = are sent every two data packets, and 17:1 for the RFC-mandated "ack = every packet" behaviour when loss is detected. Physical asymmetry ratios exceeding 20:1 are not rare. They are = increased still further if reverse traffic is present; given a nominal = 10:1 asymmetry, interleaving one ack with one MTU packet (as SFQ would) = gives an *effective* asymmetry for the downstream flow of 188:1, and = seriously affects downstream goodput. High asymmetry ratios can be tolerated by TCPs implementing sparser acks = than 1:3 ratios, as proposed in AckCC. Without AckCC however, I = understand a strict reading of the RFCs prohibits TCPs from going beyond = 1:3. Even if Linux TCPs do so anyway, billions of Windows, Mac and = mobile hosts most likely will not. This makes a solely end-to-end = solution impractical for the time being, with no obvious hope of = improvement. To maintain downstream goodput under these conditions, it is either = necessary to send bursts of acks between the upstream traffic (which is = slightly wasteful of bandwidth and also upsets RTT-sensitive TCPs like = BBR) - which Cake will do quite happily if the ack-filter is disabled - = or to delete some of the acks. When there are many upstream flows = competing with a single ack flow, the latter is the only reasonable = solution unless acks are hard-prioritised (which has negative effects of = its own, so we didn't consider it). Middleboxes *are* permitted to drop packets, and AQM routinely does so. = We found that AQM could delete acks beneficially, but that it ramped up = too slowly to really solve the problem (acks being individually small, = much larger numbers of them must be dropped once they become a = saturating flow, compared to a data flow). Also, ack flows are often = *antiresponsive*, in that dropping some of them causes an increase in = their arrival rate due to increasing the ack-clocked downstream traffic, = so it should be moderately obvious that conventional AQM strategies = don't apply. We also looked at existing ack filters' behaviour and = found it wanting, so we were initially reluctant to implement our own. However, having seen many counterexamples of how *not* to write an ack = filter, we were eventually able to write one that *didn't* break the = rules, and ensured that the information TCP relies on remained in acks = that were not deleted even when many acks were. This includes = preserving the triple-repetition rule for detecting packet loss, and = preserving the tail ack signifying reception of the end of a flow - = which naturally results in only dropping acks if it's worthwhile to do = so. In short, we're fully aware of the potential for breaking TCP this = way, and we've done our level best to avoid it. Testing revealed an appreciable, simultaneous improvement in both = downstream and upstream goodput, and in the downstream flow's internal = RTT, under appropriate traffic conditions. A more aggressive variant, = going to the edges of what might be allowed by RFCs, actually showed = *less* improvement than the standard one - it interfered with TCP's = behaviour too much. We can dig up the data if required. > Also on wifi, the queue builds in the driver queues anyway, not in the = qdisc. > So ACK filtering, if _really_ successful, would need to be = modularized. Agree that the wifi queues would also benefit from ack filtering. In = fact, with make-wifi-fast improvements active, installing Cake or any = other qdisc on that interface should be unnecessary. Cake's shaper = can't easily adapt to variable-rate links on the timescales required, = anyway. (It could if something told it directly about the variations.) However, I don't see a way to install the ack-filter as a separate = entity in its own right, without the ability to manipulate the queue = downstream of itself. The arrival of a new ack may trigger the deletion = of a previous one, never the one just arrived, yet keeping an internal = queue of acks within the filter would be counterproductive. That's why = we implemented it within Cake instead of as a separate thing. It may be possible to modularise it sufficiently that qdiscs could add = support for ack-filtering without duplicating the code, rather than the = other way around. This would also allow wifi queues to be modified the = same way. Would that be acceptable? - Jonathan Morton