From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm0-x235.google.com (mail-wm0-x235.google.com [IPv6:2a00:1450:400c:c09::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id C384C3B2A3 for ; Sat, 29 Apr 2017 06:31:06 -0400 (EDT) Received: by mail-wm0-x235.google.com with SMTP id w64so59342716wma.0 for ; Sat, 29 Apr 2017 03:31:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-transfer-encoding; bh=uKHldVFSGJhessp5AYip3Tw3u0INehv8I98ED7SnIXY=; b=K5KKheBL8juaLpHAisS78vnyZDTF5hyLCT495dcgzr3CaRuAlomY15rv4IB7jmQnYI YcocrAl4SMPMarib4M3uD/qxgRKFUm5nLzmWwEakFY3rdVMaHnql0RVY5er6YW8ANjOO o7T+ghs+mbEOTk3FhH4LINvo4ytqxhemQTSSf5rqUBaYH/XKDckXMXmVjWBO9CQ6Ig3S VELfd5B1TSocHQC7oD4k6vgkU3uOY6S8Gv7uVVoZMoAoE5HW6H9qW5dEO0XrG1vg5FqL TByFAKPJ1NtOOoKclUrFQJGY3Z/5X7GyyFj9aYn6ONUgmwV1L+239IqBzm9QrDyIE3AP KW+w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding; bh=uKHldVFSGJhessp5AYip3Tw3u0INehv8I98ED7SnIXY=; b=Enoz8NjxqcTkpqkrmuPXtbIOh8oax/hvW4UuX6zbTdje689IyiZQdWI4zCCIFQF2FD W/WoakXZpI0ArI6Vi4y8Djlg/j/CmyiL9OAr0hrnx0Q4R00TylFXCpxsAXJx2XFUZ+pI 7KbzQ0OfJEEEGz+n+kOsODT+/o1bLHFYI94J/udgz95rowJHsvKVlf1t5j9SLEmPeDhb Ni13mY6hN4PV/Ff9A5ZhAc2vnFcyurMFDGZIAJ51oMjyjgpBC8WU99Dw3MmoxJ5nOaz9 QoLoAkr4/RVRZj9RZPfNMlZfFgrnz52bsfxdjMDdHCCkraEVEJKjQLaKlbV3lrJmUWY+ JzSA== X-Gm-Message-State: AN3rC/6FnKEiTOKzcUdAPQp/5eZsnQnAoKPVlfC7y6J7dIVveXoAj9I4 +glENYypd2++T6oS X-Received: by 10.28.226.11 with SMTP id z11mr1274684wmg.91.1493461864970; Sat, 29 Apr 2017 03:31:04 -0700 (PDT) Received: from [192.168.0.3] (185.182.7.51.dyn.plus.net. [51.7.182.185]) by smtp.gmail.com with ESMTPSA id 128sm2543511wmh.21.2017.04.29.03.31.03 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 29 Apr 2017 03:31:03 -0700 (PDT) To: Jonathan Morton Cc: xnor , David Lang , Cake@lists.bufferbloat.net References: <85386ca9-f0be-f60f-796a-e5aa3b8ee212@gmail.com> <77335E95-3774-4F55-BD78-50335D811603@gmail.com> From: Andy Furniss Message-ID: Date: Sat, 29 Apr 2017 11:31:02 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:54.0) Gecko/20100101 Firefox/54.0 SeaMonkey/2.51a2 MIME-Version: 1.0 In-Reply-To: <77335E95-3774-4F55-BD78-50335D811603@gmail.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Subject: Re: [Cake] cake default target is too low for bbr? X-BeenThere: cake@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Cake - FQ_codel the next generation List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 29 Apr 2017 10:31:07 -0000 Jonathan Morton wrote: > >> On 29 Apr, 2017, at 01:26, Andy Furniss >> wrote: >> >>>>> As I understand it, increase in RTT due to queueing of >>>>> packets is the main feedback mechanism for BBR. So dropping >>>>> packets, which I already consider harmful, is really harmful >>>>> with BBR because you're not telling the sender to slow down. > > Actually, BBR considers mainly a measurement of “delivery rate”, and > paces its sending to match that. It does *not* primarily rely on a > congestion window as most TCPs do; one is provided only as a safety > net of last resort. > > Measurements of RTT are mostly used for two purposes: to size the > congestion window so that it doesn’t interfere with normal operation; > and to estimate when “queue draining” is complete after a bandwidth > probe cycle. Interesting. > >>>> If BBR does not slow down when packets are dropped, it's too >>>> hostile to use on a public network. The only way for a public >>>> network to respond to a flood of traffic higher than what it >>>> can handle is to drop packets (with a possible warning via ECN >>>> shortly before packets get dropped). If BBR doesn't slow down, >>>> it's just going to be wasting bandwidth. > >>> No it isn't. Packet loss does not equal conguestion - it never >>> did. Dropping packets to signal congestion is an ugly hack for >>> implementations that are too dumb to understand any proper >>> congestion control mechanism. >> >> Hmm, I bet a lot of carrier links are policed rather than smart >> queue. > > Policing should theoretically produce a consistent delivery rate, > which is what BBR needs to work effectively. A wrinkle here is that > most policers and shapers to date rely on a token-bucket algorithm > which permits bursts at rates well above the policy, and BBR has to > attempt to infer the policy rate from the medium-term behaviour. Ok, but it's not really going to work (be fair) when a big link with 000s of users is policed overall. Of course this shouldn't really happen - but contention exists at ISP level and local level for DOCSIS cable users. >> It also seems (OK one quick possibly flawed test), that bbr ignores >> ECN as well as drops in the sense that marked is just as high as >> dropped. > > Yes, BBR ignores ECN, which I consider to be an unfortunate feature; > it could quite reasonably be used to terminate bandwidth probes > early, before they build up a significant queue (which then needs to > be drained). Now that is unfortunate - so ECN is effectively deprecated by BBR :-( > > Cake works very well with BBR, provided it is deployed at the > upstream end of the bottleneck link. In this position, Cake happily > absorbs the temporary standing queue caused by bandwidth probes, and > the deficit-mode shaper means that BBR tends to see a consistent > delivery rate, which it considers ideal. In practice it matters > little whether the BBR sender negotiates ECN or not, in this case. > > When deployed at the downstream end of the bottleneck link, Cake > works less well with BBR - but that’s true to some extent of all > TCPs. In ingress mode, at least, dropping packets effectively causes > a reduction in the delivery rate, which should influence BBR more > strongly to correct itself. But if ECN is negotiated, these packet > drops do not occur. In both cases, the temporary standing queue > collects in the upstream dumb queue, unless Cake is configured to a > conservative enough margin below the bottleneck rate. Cake does > everything it reasonably can here, but the topology is fundamentally > unfavourable. Further testing with the aim of reducing drops seems to indicate that it's not so much target that matters but RTT. Each output two netperf (x5) runs one marked cs1, one unmarked. Sending through bulk with higher target and best effort with lower target isn't much different. Using ingress param for this test, tcp throughput is low 1.6 mbit (x5) qdisc cake 1: dev ifb0 root refcnt 2 bandwidth 16Mbit diffserv3 dual-srchost ingress rtt 100.0ms atm overhead 40 via-ethernet Sent 20770678 bytes 13819 pkt (dropped 9485, overlimits 36449 requeues 0) backlog 0b 0p requeues 0 memory used: 153Kb of 4Mb capacity estimate: 16Mbit Bulk Best Effort Voice thresh 1Mbit 16Mbit 4Mbit target 18.2ms 5.0ms 5.0ms interval 113.2ms 100.0ms 10.0ms pk_delay 13.0ms 9.6ms 0us av_delay 10.0ms 3.9ms 0us sp_delay 6us 13us 0us pkts 11654 11650 0 bytes 17565164 17563044 0 way_inds 0 0 0 way_miss 10 10 0 way_cols 0 0 0 drops 4511 4974 0 marks 0 0 0 sp_flows 4 5 0 bk_flows 2 0 0 un_flows 0 0 0 max_len 1514 1514 0 With RTT at 300ms throughput is 2.33 mbit and less drops for similar target. qdisc cake 1: dev ifb0 root refcnt 2 bandwidth 16Mbit diffserv3 dual-srchost ingress rtt 300.0ms atm overhead 40 via-ethernet Sent 31265716 bytes 20758 pkt (dropped 2563, overlimits 43619 requeues 0) backlog 0b 0p requeues 0 memory used: 153Kb of 4Mb capacity estimate: 16Mbit Bulk Best Effort Voice thresh 1Mbit 16Mbit 4Mbit target 18.2ms 15.0ms 15.0ms interval 303.2ms 300.0ms 30.0ms pk_delay 21.2ms 20.1ms 0us av_delay 18.9ms 17.0ms 0us sp_delay 4us 7us 0us pkts 11656 11665 0 bytes 17564952 17579378 0 way_inds 0 0 0 way_miss 10 10 0 way_cols 0 0 0 drops 1206 1357 0 marks 0 0 0 sp_flows 5 5 0 bk_flows 0 1 0 un_flows 0 0 0 max_len 1514 1514 0 It's even better for loss/throughput (2.44 x5) with higher rtt. qdisc cake 1: dev ifb0 root refcnt 2 bandwidth 16Mbit diffserv3 dual-srchost ingress rtt 400.0ms atm overhead 40 via-ethernet Sent 32594660 bytes 21626 pkt (dropped 1677, overlimits 44556 requeues 0) backlog 0b 0p requeues 0 memory used: 153Kb of 4Mb capacity estimate: 16Mbit Bulk Best Effort Voice thresh 1Mbit 16Mbit 4Mbit target 20.0ms 20.0ms 20.0ms interval 400.0ms 400.0ms 40.0ms pk_delay 21.9ms 20.3ms 0us av_delay 19.7ms 18.8ms 0us sp_delay 4us 4us 0us pkts 11640 11663 0 bytes 17550552 17583086 0 way_inds 0 0 0 way_miss 10 10 0 way_cols 0 0 0 drops 836 841 0 marks 0 0 0 sp_flows 5 5 0 bk_flows 0 1 0 un_flows 0 0 0 max_len 1514 1514 0 I haven't yet tried cooking up a test that includes a double queue one 10% faster fifo to see how much the latency is affected.