From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-io1-xd2a.google.com (mail-io1-xd2a.google.com [IPv6:2607:f8b0:4864:20::d2a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id C17B63CB37 for ; Fri, 26 Jul 2019 11:45:43 -0400 (EDT) Received: by mail-io1-xd2a.google.com with SMTP id e20so75307093iob.9 for ; Fri, 26 Jul 2019 08:45:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=TNxGzdEQv+DFS+gztgDOowiIe7MJXwpAobWfMgAs/SY=; b=a2uHIHKue5959pSDcsNnNOTgB15dnOfjvaKJ/xHqC83ZueMXTIxxx8co/qxhjt9EAW ASQja4KXFW2H6llFxIIb0b1nfhNmjqWtnUzB3SFExPjQ9bwqPV0tE7Jl4VU6JOnpL2ml FNUrxOgcY6tfJeSZxqwiUXI5W9lxTrMJ4wugy6WMYIMYtnS3k2jTBtZLV/tJj9OXaDVl AlRUHXAV3WYj0i3IxsRN/ziQhrGdkTQw3TfxQE1le0n3I4vZTG71xJ12fuAiBaCqbx00 sIIqX8OBqc2JSem6O+KcBy/yuKLUSqhVu7g2lHfct8vEkNLJh1OdysDyJqYA50d4UjMs st+w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=TNxGzdEQv+DFS+gztgDOowiIe7MJXwpAobWfMgAs/SY=; b=eOfwe+43tvIlfbaGXN33qPPDC29uZSqGBk4+hv9CVbJQIREdToxmadUxeBkyzfrUPU Z412hdQCRht7M4YJLt14TMtytYFbNVGhLcJRn2M8fs7OmtrpC8gKQK17gsMiBwsdVjbI xqry1ZMD8RTPYlMsZzI56ryE8cuetuGFfhi6s6apsdjaBWgveC35d5fBluoc+gMPQinA VtIXBLlGF+YxagPOkV/HsmWVo4HZQ2V6lwjAbQAk8sHXE7Zbpj6vX1KqvAjlMkm1Xq2n vWsNzmd9WfTm51m4GuJb9OBMQt75B/NG3wYCM8TBGN5Eg9X/q1zF9iIeYu3k401zydnZ A3AQ== X-Gm-Message-State: APjAAAWmbduo1iD63zYxuoQ52Iw94OI9U7UuyyfX2qh60D7ZUJ9ukqN1 D2n2pUX3ZvQ3RVLTikhwXfdOqX9N6wSrH4VuD6M= X-Google-Smtp-Source: APXvYqwfHG/+0W/JvBBSice6ethSviXRUVigKwNLRrJ2p0Tr9LDdEmBWifI2fw6emYw784SvL6zm+//G8VBMebG2AMA= X-Received: by 2002:a6b:dd17:: with SMTP id f23mr74480413ioc.213.1564155943157; Fri, 26 Jul 2019 08:45:43 -0700 (PDT) MIME-Version: 1.0 References: <364514D5-07F2-4388-A2CD-35ED1AE38405@akamai.com> <1238A446-6E05-4A55-8B3B-878C8F39FC75@gmail.com> <17B33B39-D25A-432C-9037-3A4835CCC0E1@gmail.com> <52F85CFC-B7CF-4C7A-88B8-AE0879B3CCFE@gmail.com> <87ef2myqzv.fsf@taht.net> <803D9CA8-220E-4F98-9B8E-6CE2916C3100@gmail.com> <0079BC6B-4792-48ED-90D3-D9A69407F316@gmx.de> <22af0671-fdd0-0953-fc96-55b34beb0be9@bobbriscoe.net> <3EB0D59D-69A7-4730-BCDF-10E5C61EF987@heistp.net> In-Reply-To: From: Dave Taht Date: Fri, 26 Jul 2019 08:45:30 -0700 Message-ID: To: Neal Cardwell Cc: Pete Heist , "De Schepper, Koen (Nokia - BE/Antwerp)" , "ecn-sane@lists.bufferbloat.net" , "tsvwg@ietf.org" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Subject: Re: [Ecn-sane] The state of l4s, bbrv2, sce? X-BeenThere: ecn-sane@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion of explicit congestion notification's impact on the Internet List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Jul 2019 15:45:43 -0000 On Fri, Jul 26, 2019 at 8:37 AM Neal Cardwell wrote: > > On Fri, Jul 26, 2019 at 11:05 AM Dave Taht wrote: >> >> 1) BBRv2 is now available for public hacking. I had a good readthrough >> last night. >> >> The published tree applies cleanly (with a small patch) to net-next. >> I've had a chance to read through the code (lots of good changes to >> bbr!). >> >> Although neal was careful to say in iccrg the optional ecn mode uses >> "dctcp/l4s-style signalling", he did not identify how that was >> actually applied >> at the middleboxes, and the supplied test scripts >> (gtests/net/tcp/bbr/nsperf) don't do that. All we know is that it's >> set to kick in at 20 packets. Is it fq_codel's ce_threshold? red? pie? >> dualpi? Does it revert to drop on overload? > > > As mentioned in the ICCRG session, the TCP source tree includes the scrip= ts used to run the tests and generate the graphs in the slide deck. Here is= the commit I was mentioning: > > https://github.com/google/bbr/commit/e76d4f89b0c42d5409d34c48ee6f8d324= 07d4b8d > > So you can look at exactly how each test was run, and re-run those tests = yourself, with the v2alpha code or any experimental tweaks you might make b= eyond that. > > To answer your particular question, the ECN marks were from a bottleneck = qdisc configured as: > > codel ce_threshold 242us limit 1000 target 100ms thx neal! I missed that! > I'm not claiming that's necessarily the best mechanism or set of paramete= rs to set ECN marks. The 20-packet number comes from the DCTCP SIGCOMM 2010= paper's recommendation for 1Gbps bottlenecks. I just picked this kind of a= pproach because the bare metal router/switch hardware varies, so this is a = simple and easy way for everyone to experiment with the exact same ECN sett= ings. ok! > >> Is it running on bare metal? 260us is at the bare bottom of what linux >> can schedule reliably, vms are much worse. > > > I have tried both VMs and bare metal with those scripts, and of course th= e VMs are quite noisy and the bare metal results much less noisy. So the gr= aphs are from runs on bare metal x86 server-class machines. Good to know. On the cloud I use (linode) 1ms was the best I could hope for, and even then, dang jittery. (it was much worse 8 years back when xen underneath could be in the 10-20ms range!). There are major jitter issues on lower end hardware but I don't know how bad they are post spectre fixes, been afraid to look. containers are a huge improvement over vms but still break things like tsq. > > neal > --=20 Dave T=C3=A4ht CTO, TekLibre, LLC http://www.teklibre.com Tel: 1-831-205-9740