From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from uplift.swm.pp.se (swm.pp.se [212.247.200.143]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 2E4623CB35; Thu, 14 Dec 2017 03:22:22 -0500 (EST) Received: by uplift.swm.pp.se (Postfix, from userid 501) id 7785EB3; Thu, 14 Dec 2017 09:22:20 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=swm.pp.se; s=mail; t=1513239740; bh=EjHmYjYdB5ezixO1K41uyh/kuPkBnie2zCCcfYL8qjQ=; h=Date:From:To:cc:Subject:In-Reply-To:References:From; b=FGjxBE5/UKFgkdoCs/7C9QQodsrWydtW4MqLYywPFCUWGvTQUA1RqaI2dGu4jVn3E +gD4xV+5Ir/Ha7RoNmin/ALmPn6l8ls+e6dqigt8eenEmhP3bs8s6pDqjWroKAHoFg Jxu8SHbyVLSbnrlM39zIwxMTvM/mdS1xAw5LPojg= Received: from localhost (localhost [127.0.0.1]) by uplift.swm.pp.se (Postfix) with ESMTP id 5FFD1B1; Thu, 14 Dec 2017 09:22:20 +0100 (CET) Date: Thu, 14 Dec 2017 09:22:20 +0100 (CET) From: Mikael Abrahamsson To: Jonathan Morton cc: cerowrt-devel@lists.bufferbloat.net, bloat In-Reply-To: Message-ID: References: <1512417597.091724124@apps.rackspace.com> <87wp1rbxo8.fsf@nemesis.taht.net> <1513119230.638732339@apps.rackspace.com> <7D300E07-536C-4ABD-AE38-DDBAF30E80D7@pnsol.com> User-Agent: Alpine 2.20 (DEB 67 2015-01-07) Organization: People's Front Against WWW MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed Subject: Re: [Cerowrt-devel] [Bloat] DC behaviors today X-BeenThere: cerowrt-devel@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: Development issues regarding the cerowrt test router project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Dec 2017 08:22:22 -0000 On Wed, 13 Dec 2017, Jonathan Morton wrote: > Ten times average demand estimated at time of deployment, and struggling > badly with peak demand a decade later, yes. And this is the > transportation industry, where a decade is a *short* time - like less > than a year in telecoms. I've worked in ISPs since 1999 or so. I've been at startups and I've been at established ISPs. It's kind of an S curve when it comes to traffic growth, when you're adding customers you can easily see 100%-300% growth per year (or more). Then after market becomes saturated growth comes from per-customer increased usage, and for the past 20 years or so, this has been in the neighbourhood of 20-30% per year. Running a network that congests parts of the day, it's hard to tell what "Quality of Experience" your customers will have. I've heard of horror stories from the 90ties where a then large US ISP was running an OC3 (155 megabit/s) full most of the day. So someone said "oh, we need to upgrade this", and after a while, they did, to 2xOC3. Great, right? No, after that upgrade both OC3:s were completely congested. Ok, then upgrade to OC12 (622 megabit/s). After that upgrade, evidently that link was not congested a few hours of the day, and of course needed more upgrades. So at the places I've been, I've advocated for planning rules that say that when the link is peaking at 5 minute averages of more than 50% of link capacity, then upgrade needs to be ordered. This 50% number can be larger if the link aggregates larger number of customers, because typically your "statistical overbooking" varies less the more customers participates. These devices do not do per-flow anything. They might have 10G or 100G link to/from it with many many millions of flows, and it's all NPU forwarding. Typically they might do DIFFserv-based queueing and WRED to mitigate excessive buffering. Today, they typically don't even do ECN marking (which I have advocated for, but there is not much support from other ISPs in this mission). Now, on the customer access line it's a completely different matter. Typically people build with BRAS or similar, where (tens of) thousands of customers might sit on a (very expensive) access card with hundreds of thousands of queues per NPU. This still leaves just a few queues per customer, unfortunately. So these do not do per-flow anything either. This is where PIE comes in, because these devices like these can do PIE in the NPU fairly easily because it's kind of like WRED. So back to the capacity issue. Since these devices typically aren't good at assuring per-customer access to the shared medium (backbone links), it's easier to just make sure the backbone links are not regularily full. This doesn't mean you're going to have 10x capacity all the time, it probably means you're going to be bouncing between 25-70% utilization of your links (for the normal case, because you need spare capacity to handle events that increase traffic temporarily, plus handle loss of capacity in case of a link fault). The upgrade might be to add another link, or a higher tier speed interface, bringing down the utilization to typically half or quarter of what you had before. -- Mikael Abrahamsson email: swmike@swm.pp.se