From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm0-x235.google.com (mail-wm0-x235.google.com [IPv6:2a00:1450:400c:c09::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 901EB3B2A4 for ; Mon, 8 May 2017 06:37:11 -0400 (EDT) Received: by mail-wm0-x235.google.com with SMTP id b84so49501956wmh.0 for ; Mon, 08 May 2017 03:37:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-transfer-encoding; bh=f0XocOyGOLAukmprIK7fHrXPzlS5RnkvoPM9ldelObA=; b=ObZ+mzRwSHPENzibr5CXFVNXq9aP/32KqR6eeHHJKOL9NyRXXrGA0MiXI0AznIXQIg 0j0DfQn7T0sUuMnoMAFJE4mOoKYOrw/mopr5xX8ogdAUQD6RRDaTrFvjmdYrs/+rWzyX oCwHZMM8in4MuCF5yGI/ydX7OCDytyWIqVNKC/jtGEX8og0tnaQGqZSoBmFvAyvRFn0l mgivIzom0THGsI8WLV//yXhc9x1WXCx++M8DzvZ+SotjOYNC+1lC8vOgydfzlVTb2cWm 4GAb6OcZWjFVPSuBJ+H0RYjWlnvByApvOUJ09rGFJMyBjTwN9aOgycovYz8OjS9V+2fM lQMQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding; bh=f0XocOyGOLAukmprIK7fHrXPzlS5RnkvoPM9ldelObA=; b=DlHHcNSXCe5fdSr1sHs3pM6vejQxRd/ALzuKozl+Gx+8vGee64L6JZYnqU72XQRBvm OI2Q+tWDDBTP2aENjfxwpToSbWkfyQ2zUqoQvVvkOzSo+Hbo2HPnN3CSaMYDKMxiQ0tl mclMvSgzq6DL651L29FOVUWQw3R6yh7ct95HwIc9HF8Re9ITPSL7EJHnW0HQG1qsplAW l8GgT4UlinitL3HXZnbgn+5sXFNFQ6Er1dbJxPpQUXYfLcWNEcZ1Fl/NJSo2KBkCyJdJ J9c8Xc3pbvFeK09puuVT3nZTpcL48hBDRDTB9PI6xS4mV5I2wpO8U5LpPV5M/JIL3ByX KA0g== X-Gm-Message-State: AN3rC/4xLlzc9sqUt0q/7MdqAtDnqd9kXT1v9ewGJSIdTk2ut0OEi1A/ riPnYLxcsAnMtw== X-Received: by 10.28.150.86 with SMTP id y83mr12622768wmd.46.1494239830533; Mon, 08 May 2017 03:37:10 -0700 (PDT) Received: from [192.168.0.3] (185.182.7.51.dyn.plus.net. [51.7.182.185]) by smtp.gmail.com with ESMTPSA id y63sm16095421wme.31.2017.05.08.03.37.09 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 08 May 2017 03:37:09 -0700 (PDT) To: Jim Gettys Cc: xnor , David Lang , Cake@lists.bufferbloat.net References: <85386ca9-f0be-f60f-796a-e5aa3b8ee212@gmail.com> <5ca09d5c-e674-110a-72e4-b8fd434c7a5d@gmail.com> <5e76ed9f-7eae-f309-2d64-3ed34f19d3d4@gmail.com> From: Andy Furniss Message-ID: Date: Mon, 8 May 2017 11:37:08 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:54.0) Gecko/20100101 Firefox/54.0 SeaMonkey/2.51a2 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Subject: Re: [Cake] cake default target is too low for bbr? X-BeenThere: cake@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Cake - FQ_codel the next generation List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 May 2017 10:37:11 -0000 Jim Gettys wrote: > On Thu, May 4, 2017 at 6:22 AM, Andy Furniss wrote: > >> Jim Gettys wrote: >> >>> On Wed, May 3, 2017 at 5:50 AM, Andy Furniss >>> wrote: >>> >>> Andy Furniss wrote: >>>> >>>> Andy Furniss wrote: >>>>> >>>>> b) it reacts to increase in RTT. An experiment with 10 Mbps >>>>> bottleneck, >>>>> >>>>>> 40 ms RTT and a typical 1000 packet buffer, increase in RTT >>>>>>> with BBR is ~3 ms while with cubic it is over 1000 ms. >>>>>>> >>>>>>> >>>>>> That is a nice aspect (though at 60mbit hfsc + 80ms bfifo I >>>>>> tested with 5 tcps it was IIRC 20ms vs 80 for cubic). I >>>>>> deliberately test using ifb on my PC because I want to pretend >>>>>> to be a router - IME (OK it was a while ago) testing on eth >>>>>> directly gives different results - like the locally generated >>>>>> tcp is backing off and giving different results. >>>>>> >>>>>> >>>>> I retested this with 40ms latency (netem) with hfsc + 1000 pfifo >>>>> on ifb. >>>>> >>>>> >>>> So, as Jonathan pointed out to me in another thread bbr needs fq >>>> and it seems fq only wotks on root of a real eth, which means thay >>>> are invalid tests. >>>> >>>> >>> ​Specifically, BBR needs packet pacing to work properly: the >>> algorithm depends on the packets being properly paced. >>> >>> Today, fq is the only qdisc supporting pacing. >>> >>> The right answer would be to add packet pacing to cake/fq_codel >>> directly. Until that is done, we don't know how BBR will work in our >>> world. - Jim​ >>> >> >> I guess you mean so cake could be used on egress of sender (in place of >> fq)? >> > > ​Yes. > ​ > >> >> That's not really the test that I intend to do, which is more like - >> >> [boxA bbr+fq] -> [boxB simulate ISP buffer] -> [boxC cake ingress shape] >> a bit lower than "line" rate and see how much "ISP" buffer gets filled. >> >> Also compare bbr, cubic and netem different rtts etc. > > > ​Ok. The usual warnings about netem being dangerous apply, though netem > can be useful if run on a separate machine. Netem is an attractive > nuisance, but has caused lots of results to be ultimately useless.... Be > careful. > - Jim Yea, I saw the warning about netem on the website - tricky as I would need 4 boxes to really isolate it, and I've only got three, so it's not ideal. I tested with it delaying acks on ingress of sender which had fq on egress. Also did some tcpdumps with different setups to see if it was clumping acks - it seemed smooth. With this setup my results are much the same as before = bbr is harder to shape on ingress vs cubic, the longer the rtt to sender the worse it is. I was testing 18mbit dsl sim with mainly 16mbit ingress. Repeatedly grabbing 1 or 2 meg files (like dash) spikes latency every time. With rtt of 40ms unshaped it will burst to 80+ms, 16mbit behind 18mbit spikes 50ms. IIRC to get spikes below 20ms I needed 12mbit = too low. For continuous downloads, after initial spike a single tcp wasn't too bad, even 5 can be controlled latency wise at 16mbit. The problem with 5 vs 1 is the number of drops, 1 not too bad, but 5 will drop 1/10 so tcp ends up sacking almost per packet, which is not good for those with limited upstream. ECN marks more packets than would be dropped, almost all in the 5 case so also causes many upstream acks. This is with cakes ingress parameter and default rtt of 100ms. Changing to 300ms does reduce the drops/marks as in my previous post, but it makes controlling the startup spikes slightly less effective - though without going really low they weren't controlled very well anyway.