From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-io1-xd2a.google.com (mail-io1-xd2a.google.com [IPv6:2607:f8b0:4864:20::d2a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id CAF543B2A4 for ; Thu, 29 Aug 2019 15:10:55 -0400 (EDT) Received: by mail-io1-xd2a.google.com with SMTP id s21so9205898ioa.1 for ; Thu, 29 Aug 2019 12:10:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=yTGXdEg1ZaQuJUKp3QOva4u2LDyEzUjasErA+kmT35c=; b=WQ2PKbiYKiqWqf1TD3/Afd7ulyN2BiWBoxSj0TFtq9/EFDyxUxwa7WGNYhF0yC2F7L XvFj68ALNOZlV6cG4E+7xZiISgi4GanjG4eA19rMlq0NxOnLx0+/G2X5nrWUk/5Piz8k LmonYbTzcfAmaLPoeoIAm1GdgY6BmJjdLEk3IH58SXpt8b9D5HNOMBpK41id5wzxsvly IfsdmQt+/JUjrEQYaOE8mr5J8KJJETRKULCGhnBnKRRXI5xZ1magqfKV6EHwVAvhWNQQ UaRucwuUP84u6uCXfRqCmbi1oOBmG7cS4wVLsu+XtigfLQo/maZoj6TXtqp1I7Za6uyv qTww== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=yTGXdEg1ZaQuJUKp3QOva4u2LDyEzUjasErA+kmT35c=; b=uXB0DO8Dxjk6iV8rHAXDEPXbHtFCtSUJHRTjaG7n42OaoH1tmQG6bxwZ1dpjr+20J2 NfVnRnHdOFyCLbz2JQnm3VG+eEl1G3g/TAtwd52mGHqqszHj9o2y8Zl8I0mwfSBq7sU6 nwFamdudTH1n5QWm+VBWJoCXpXsDOjOjxfpqCyrdvQj0Hf9cRLkyPTQky6m0TX9n6gc/ QYJ9BOaQPE5XfRw5O6A9IZ2naoyp/L25YoCoJQrZl8nKr56jgJrEtkoIcedosryvsniL Tell5QZOm7T9yIFAoNAMu8oss0eswP0UyxcdLL2vipQrtBm3YWzKVpc2hY09XF6xYTFY n6VA== X-Gm-Message-State: APjAAAWfeVYNTzGEevDJpKq2s4FiVn89WWaUJjHJswMYToRoqd+67/Jj ks4WiBfwmU0ROe90AmeFGyWZvmealJ1S2Nw0X5Q= X-Google-Smtp-Source: APXvYqzp2g9wn6FirtfcVDXB11HdjtWoMPHdwQXtQR2Zn4GPleK/Mcl+0bv/a2uVJO1AMhJR7M8QwA5lWSI3Mesb9GY= X-Received: by 2002:a02:b713:: with SMTP id g19mr1097419jam.77.1567105855033; Thu, 29 Aug 2019 12:10:55 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Dave Taht Date: Thu, 29 Aug 2019 12:10:43 -0700 Message-ID: To: Jonathan Morton Cc: ECN-Sane Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Subject: Re: [Ecn-sane] rfc3168 sec 6.1.2 X-BeenThere: ecn-sane@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion of explicit congestion notification's impact on the Internet List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Aug 2019 19:10:55 -0000 On Thu, Aug 29, 2019 at 7:42 AM Jonathan Morton wro= te: > > > On 29 Aug, 2019, at 4:51 pm, Dave Taht wrote: > > > > I am leveraging hazy memories of old work a years back where I pounded = 50 ? 100 ? flows through a 100Mbit ethernet > > At 100 flows, that gives you 1Mbps per flow fair share, so 80pps or 12.5m= s between packets on each flow, assuming they're all saturating. This also= means you have a minimum sojourn time (for saturating flows) of 12.5ms, wh= ich is well above the Codel target, so Codel will always be in dropping-sta= te and will continuously ramp up its signalling frequency (unless some miti= gation is in place for this very situation, which there is in Cake). > > Both Cake and fq_codel should still be able to prioritise sparse flows to= sub-millisecond delays under these conditions. They'll be pretty strict a= bout what counts as "sparse" though. Your individual keystrokes and echoes= should get through quickly, but output from programs may end up waiting. > > > A) fq_codel with drop had MUCH lower RTTs - and would trigger RTOs etc > > RTOs are bad. They indicate that the steady flow of traffic has broken d= own on that flow due to tail loss, which is a particular danger at very sma= ll cwnds. They indicated that traffic has broken down for any of a zillion reasons. RTO's for example, are what gets tcp restarted after babel does the circuit breaker thing on this test and restores it. RTOs are Good. :) > Cake tries to avoid them by not dropping the last queued packet from any = given flow. Fq_codel doesn't have that protection, so in non-ECN mode it w= ill drop way too many packets in a desperate (and misguided) attempt to mai= ntain the target sojourn time. We are trying to encourage others to stop editorizing so much. As the author of this behavior in fq_codel, my reasoning at the time was that under conditions of overload that there were usually packets "in the network", and keeping the last packet in the queue scaled badly in terms of total RTT. Saying "go away, come back later" was a totally reasonable response, baked into TCPs since the very beginning. I'm glad that cake and fq_codel have a different response curve here. It's interesting. Catagorizing the differences between approaches is good. As best as I can recall I put this behavior into fq_codel after some very similar testing back in 2012. > What you need to understand here is that dropped packets increase *applic= ation* latency, even if they also reduce the delay to individual packets. = ECN doesn't incur that problem. Well, let me point at my data here: http://blog.cerowrt.org/post/ecn_fq_codel_wifi_airbook/ We need to be clear about what we consider an "application". I tend to think about things more as "human facing" or not, and optimize for humans first. In this case dropped packets on a 2 second flow account for a maximum of 16ms increase for FCT. Inperceptable. Compared to making room for other packets from other flows at the point of contention is a win for those other flows. In particular (and perhaps we can show this with a heavy load test) having shorter RTTs from drop makes it faster for a new or existing flows to grab back bandwidth when part of that load exits. I've long bought the argument for human interactive flows that need a reliable transport - that ecn is good - as we did in mosh. But (being chicken) on doing it to everything, not so much. Anyway, the cwnd 1 + retransmit (or pacing!) idea would hopefully reduce the ecn'd RTTs to something more comparable to the drop in this particular test, which would be a step forward. I'll get to your other points below, later. > > B) cake (or fq_codel with ecn) hit, I don't remember, 40ms tcp delays. > > A delay of 40ms suggests about 3 packets per flow are in the queue. That= 's pretty close to the minimum cwnd of 2. One would like to do better than= that, of course, but options for doing so become limited. > > I would expect SCE to do better at staying *at* the minimum cwnd in these= conditions. That by itself would reduce your delay to 25ms. Combined wit= h setting the CA pacing scale factor to 40%, that would also reduce the ave= rage packets per flow in the queue to 0.8. I think that's independent of w= hether the receiver still acks only every other segment. The delay on each= flow would probably go down to about 10ms on average, but I'm not going to= claim anything about the variance around that value. > > Since 10ms is still well above the normal Codel target, SCE will be signa= lling 100% to these flows, and thus preventing them from increasing the cwn= d from 2. > > > C) The workload was such that the babel protocol (1000? routes - 4 > > packet non-ecn'd udp bursts) would eventually fail - dramatically, by > > retracting the route I was on and thus acting as a circuit breaker on > > all traffic, so I'd lose connectivit for 16 sec > > That's a problem with Babel, not with ECN. A robust routing protocol sho= uld not drop the last working route to any node, just because the link gets= congested. It *may* consider that link as non-preferred and seek alternat= ive routes that are less congested, but it *must* keep the route open (if i= t is working at all) until such an alternative is found. > > But you did find that turning on ECN for the routing protocol helped. So= the problem wasn't latency per se, but packet loss from the AQM over-react= ing to that latency. > > > Anyway, 100 flows, no delays, straight ethernet, and babel with 1000+ r= outes is easy to setup as a std test, and I'd love it if y'all could have t= hat in your testbed. > > Let's put it on the todo list. Do you have a working script we can just = use? > > - Jonathan Morton --=20 Dave T=C3=A4ht CTO, TekLibre, LLC http://www.teklibre.com Tel: 1-831-205-9740