From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-lf0-x231.google.com (mail-lf0-x231.google.com [IPv6:2a00:1450:4010:c07::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.bufferbloat.net (Postfix) with ESMTPS id 849EC3B25E; Mon, 3 Oct 2016 06:00:21 -0400 (EDT) Received: by mail-lf0-x231.google.com with SMTP id l131so150375224lfl.2; Mon, 03 Oct 2016 03:00:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:subject:from:in-reply-to:date:cc:message-id:references :to; bh=dJCUnRckqke9bFsKDlT561ZU/EoXL5ZAAU1Gz6/n6XI=; b=SdqsaaJyck8lbQPcfdKUjXGQP3nq8zYzmxh8ZsuvEpVddHgvR4MwbAY1GyjAs3qWCS FnRMoriJniZh1YmVeBUHWMBu+QVa4z5bgbaKXnk+KKUzAS91h4dmkfhkvbHB6gpa+wCN E/Xgr4EHYsr12KItlJz55ZctyYIuVtb7RvBXSjPeQucwnmzVGUtwsQX6C8KtXmp3qScJ EdaCZSwU5T3qKLs3KnzsvRV77r7vRStMp49DnE2iE84fIta6lO8ybEFEQqBn7sikyDqv 0Ux6XlR3rI8cpUivI0Bicnpy0k/BiMVLtt35RCpm4IIEqUOVUBm0ZqDacMSwCjKyM5w0 5PaA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :message-id:references:to; bh=dJCUnRckqke9bFsKDlT561ZU/EoXL5ZAAU1Gz6/n6XI=; b=VLCH55gCeMHm19aNqdskXTjpFyEgoFUBOurxtkIQamF7g039uADTsXCUZuqsw9Gant 68L2VN83LjuTBaVJi2TVA1swQuvOoDO3qahVzEsPi5+o78LUn0ZbDwl1KoXIg+8fAuRr DDOq4p9z/vlbOlCsFlsg5pDBvi968ueiWwloKf5M3Fs+dO/SpROAxwI6tHnI9+pS0zcz ENlM5Q2gn9pMsOh9U31rVfwsgr+42fOeEWQxyZDGQyCW8sI2Rqwm8nLrvo9PKLOabluu H8g+USMxTdw8GxNVC+xcOhsplmSGvEyn0nPf7NEodDO0ejFntpJZmDnyAzHtQpnW+pz6 TS1A== X-Gm-Message-State: AA6/9RlBPRkiLQIUR8DFDqQM2RqjTiqJGLFOo9miFu655JHKY5SQmaGXCl8/0wOnIxBP1Q== X-Received: by 10.194.112.131 with SMTP id iq3mr16435349wjb.123.1475488820120; Mon, 03 Oct 2016 03:00:20 -0700 (PDT) Received: from [192.168.72.54] (h-1169.lbcfree.net. [185.99.119.68]) by smtp.gmail.com with ESMTPSA id n131sm18096335wmd.3.2016.10.03.03.00.19 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 03 Oct 2016 03:00:19 -0700 (PDT) Content-Type: multipart/alternative; boundary="Apple-Mail=_60A161E5-41CB-4569-80C4-A74EB99BC45A" Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) From: Phineas Gage In-Reply-To: Date: Mon, 3 Oct 2016 12:00:18 +0200 Cc: codel@lists.bufferbloat.net, make-wifi-fast@lists.bufferbloat.net Message-Id: References: <945ED215-49E0-4F56-8B9A-FA95C0A82ABE@gmail.com> <75B6F546-894B-4257-95F6-5329273B07D1@gmail.com> To: Jonathan Morton X-Mailer: Apple Mail (2.3124) Subject: Re: [Codel] Using fq_codel with a WiFi uplink to the Internet X-BeenThere: codel@lists.bufferbloat.net X-Mailman-Version: 2.1.20 Precedence: list List-Id: CoDel AQM discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Oct 2016 10:00:21 -0000 --Apple-Mail=_60A161E5-41CB-4569-80C4-A74EB99BC45A Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On Sep 23, 2016, at 1:54 PM, Phineas Gage = wrote: >=20 >>> Option A: We can choose a maximum of 40/40 Mbit with 1:5 = aggregation, meaning we could get 40 Mbit, but we could also get a lot = less at times (8 Mbit I assume), depending on other network load. >>=20 >> The 1:5 aggregation is worth exploring a bit more deeply. Usually = this refers to the ratio between the total bandwidth provisioned across = all customers (in some region and some service class) and the minimum = backhaul capacity within and at the far edge of the ISP=E2=80=99s = network. The theory is that customers tend not to use all of their = bandwidth all of the time, though they do tend to use all of it some of = the time. This theory works fairly well in practice as long as the = traffic patterns are not too correlated and are distributed over a = sufficient number of customers. >>=20 >> In order to be dragged down to 8Mbps, you would have to see all the = other customers sharing your backhaul trying to use 8Mbps or more (on = average) at the same time. This is unlikely to occur often; consider = major sportsball championship final matches, or national disasters such = as one we recently had an anniversary of, as the most likely triggers. = Under such circumstances, you=E2=80=99ll need to step in and manually = experiment to find out what works, but shaping at 8Mbps would be a = reasonable default reaction. >>=20 >> Of more concern to you is how much external load is needed to pull = your share of the backhaul significantly below 40Mbps - or, = alternatively, below the 20Mbps you can get with the more expensive = uncontended service. This will vary depending on how many customers the = 5:1 average is spread over - you can=E2=80=99t actually calculate it = from the information given. >>=20 >> So the question is whether they have a 40/40 backhaul shared among 5 = customers, or a 1G/1G backhaul which they plan to share among up to 125 = customers, each with 40/40 service. I think the latter is a perfectly = reasonable possibility, and would be much more robust than the former = option which would give you a reduction in throughput as soon as *any* = of the other customers on the same backhaul segment did *anything*. >>=20 >> It=E2=80=99s a question worth asking your WISP. Be ready to point = out that you don=E2=80=99t plan to actually *use* 40Mbps full-time, but = that your AQM solution depends on knowing how much bandwidth is = available for when *your* customers randomly demand it. >=20 > Thanks a lot for this great information, I=E2=80=99ll ask them more = about what=E2=80=99s behind their aggregation ratio. As I mentioned in = one reply earlier, I may ask for flexibility in our contract, try the = aggregated service, then switch to guaranteed service depending on the = actual performance of their network. If I can rate limit to 30/30 and = fq_codel, and it=E2=80=99s 99% effective, I=E2=80=99d rather take that = than settle for a guaranteed 20/20 on-season and 6/6 off-season to get = to the same annual contract price. Following up on the WISP results, I tested the 40 Mbps non-guaranteed = connection for a week or so, and the truth is that the speed varies much = more than expected. Although I once saw a 40 Mbps download, that=E2=80=99s= more of a black swan. Download throughput during Internet rush hour = regularly dropped to 5-10 Mbps, and I once saw 1 Mbps. Upload throughput = stays higher. But I can=E2=80=99t see how =E2=80=9Caggregation 1:5=E2=80=9D= has any meaning for me, as I saw speeds below 8 Mbps. There is probably = no way I can effectively control either the upload or download queues = effectively with this type of service. Also, they seem to prioritize traffic to speedtest.net, because while = the graphs to those servers looked pretty good and consistent (I = scripted a test every 30 minutes using speedtest-cli), throughput = reported by using wget to various high-powered Ubuntu servers around us = in Europe told a different story. During the day, 25-30 Mbps was = routine, and in the evening, 5-10 Mbps was routine, with outliers on = either end. This was of course with a cabled client and no other traffic = from us on the connection. So I=E2=80=99ll be finding out more details = about the =E2=80=9Cguaranteed=E2=80=9D 10 Mbps service. Also a couple other comments to anyone interested: - I saw a post about recent work getting Cake to run on EdgeOS, so I = expressed interest in that for the EdgeRouter X, which I installed = yesterday: = http://community.ubnt.com/t5/EdgeMAX/Cake-compiled-for-the-ERL/td-p/167984= 4 - I might make it to the OpenWrt summit (http://openwrtsummit.org/) in = Berlin next Thursday, 10/13, in case I catch anyone there. --Apple-Mail=_60A161E5-41CB-4569-80C4-A74EB99BC45A Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8
On = Sep 23, 2016, at 1:54 PM, Phineas Gage <phineas919@gmail.com> wrote:

Option A: We can choose = a maximum of 40/40 Mbit with 1:5 aggregation, meaning we could get 40 = Mbit, but we could also get a lot less at times (8 Mbit I assume), = depending on other network load.

The 1:5 aggregation is worth exploring a bit more deeply. =  Usually this refers to the ratio between the total bandwidth = provisioned across all customers (in some region and some service class) = and the minimum backhaul capacity within and at the far edge of the = ISP=E2=80=99s network.  The theory is that customers tend not to = use all of their bandwidth all of the time, though they do tend to use = all of it some of the time.  This theory works fairly well in = practice as long as the traffic patterns are not too correlated and are = distributed over a sufficient number of customers.

In order to be dragged down to 8Mbps, you would have to see = all the other customers sharing your backhaul trying to use 8Mbps or = more (on average) at the same time.  This is unlikely to occur = often; consider major sportsball championship final matches, or national = disasters such as one we recently had an anniversary of, as the most = likely triggers.  Under such circumstances, you=E2=80=99ll need to = step in and manually experiment to find out what works, but shaping at = 8Mbps would be a reasonable default reaction.

Of more concern to you is how much external load is needed to = pull your share of the backhaul significantly below 40Mbps - or, = alternatively, below the 20Mbps you can get with the more expensive = uncontended service.  This will vary depending on how many = customers the 5:1 average is spread over - you can=E2=80=99t actually = calculate it from the information given.

So = the question is whether they have a 40/40 backhaul shared among 5 = customers, or a 1G/1G backhaul which they plan to share among up to 125 = customers, each with 40/40 service.  I think the latter is a = perfectly reasonable possibility, and would be much more robust than the = former option which would give you a reduction in throughput as soon as = *any* of the other customers on the same backhaul segment did = *anything*.

It=E2=80=99s a question worth = asking your WISP.  Be ready to point out that you don=E2=80=99t = plan to actually *use* 40Mbps full-time, but that your AQM solution = depends on knowing how much bandwidth is available for when *your* = customers randomly demand it.

Thanks a lot for = this great information, I=E2=80=99ll ask them more about what=E2=80=99s = behind their aggregation ratio. As I mentioned in one reply earlier, I = may ask for flexibility in our contract, try the aggregated service, = then switch to guaranteed service depending on the actual performance of = their network. If I can rate limit to 30/30 and fq_codel, and it=E2=80=99s= 99% effective, I=E2=80=99d rather take that than settle for a = guaranteed 20/20 on-season and 6/6 off-season to get to the same annual = contract price.

Following up on the WISP results, I tested the 40 Mbps = non-guaranteed connection for a week or so, and the truth is that the = speed varies much more than expected. Although I once saw a 40 Mbps = download, that=E2=80=99s more of a black swan. Download throughput = during Internet rush hour regularly dropped to 5-10 Mbps, and I once saw = 1 Mbps. Upload throughput stays higher. But I can=E2=80=99t see how = =E2=80=9Caggregation 1:5=E2=80=9D has any meaning for me, as I saw = speeds below 8 Mbps. There is probably no way I can effectively control = either the upload or download queues effectively with this type of = service.

Also, = they seem to prioritize traffic to speedtest.net, because while the graphs to those servers = looked pretty good and consistent (I scripted a test every 30 minutes = using speedtest-cli), throughput reported by using wget to various = high-powered Ubuntu servers around us in Europe told a different story. = During the day, 25-30 Mbps was routine, and in the evening, 5-10 Mbps = was routine, with outliers on either end. This was of course with a = cabled client and no other traffic from us on the connection. So I=E2=80=99= ll be finding out more details about the =E2=80=9Cguaranteed=E2=80=9D 10 = Mbps service.

Also a couple other comments to anyone interested:

- I saw a post about = recent work getting Cake to run on EdgeOS, so I expressed interest in = that for the EdgeRouter X, which I installed yesterday: http://community.ubnt.com/t5/EdgeMAX/Cake-compiled-for-the-ERL/= td-p/1679844

- I might make it to the OpenWrt summit (http://openwrtsummit.org/) in Berlin next Thursday, = 10/13, in case I catch anyone there.

= --Apple-Mail=_60A161E5-41CB-4569-80C4-A74EB99BC45A--