From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp105.iad3a.emailsrvr.com (smtp105.iad3a.emailsrvr.com [173.203.187.105]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by huchra.bufferbloat.net (Postfix) with ESMTPS id 233BD21F33E for ; Thu, 19 Mar 2015 13:29:22 -0700 (PDT) Received: from smtp6.relay.iad3a.emailsrvr.com (localhost.localdomain [127.0.0.1]) by smtp6.relay.iad3a.emailsrvr.com (SMTP Server) with ESMTP id 8AF60180246; Thu, 19 Mar 2015 16:29:21 -0400 (EDT) Received: from app58.wa-webapps.iad3a (relay-webapps.rsapps.net [172.27.255.140]) by smtp6.relay.iad3a.emailsrvr.com (SMTP Server) with ESMTP id 46A551802EC; Thu, 19 Mar 2015 16:29:21 -0400 (EDT) X-Sender-Id: dpreed@reed.com Received: from app58.wa-webapps.iad3a (relay-webapps.rsapps.net [172.27.255.140]) by 0.0.0.0:25 (trex/5.4.2); Thu, 19 Mar 2015 20:29:21 GMT Received: from reed.com (localhost.localdomain [127.0.0.1]) by app58.wa-webapps.iad3a (Postfix) with ESMTP id 30049380061; Thu, 19 Mar 2015 16:29:21 -0400 (EDT) Received: by apps.rackspace.com (Authenticated sender: dpreed@reed.com, from: dpreed@reed.com) with HTTP; Thu, 19 Mar 2015 16:29:21 -0400 (EDT) Date: Thu, 19 Mar 2015 16:29:21 -0400 (EDT) From: dpreed@reed.com To: "Livingood, Jason" MIME-Version: 1.0 Content-Type: text/plain;charset=UTF-8 Content-Transfer-Encoding: quoted-printable Importance: Normal X-Priority: 3 (Normal) X-Type: plain In-Reply-To: References: <20150316203532.05BD21E2@taggart.lackof.org> <123130.1426635142@turing-police.cc.vt.edu> <15A0911A-E3B7-440A-A26B-C5E1489EA98B@viagenie.ca> <1426773234.362612992@apps.rackspace.com> X-Auth-ID: dpreed@reed.com Message-ID: <1426796961.194223197@apps.rackspace.com> X-Mailer: webmail/11.3.13-RC Cc: "cerowrt-devel@lists.bufferbloat.net" , bloat Subject: Re: [Cerowrt-devel] =?utf-8?q?=5BBloat=5D__DOCSIS_3+_recommendation?= =?utf-8?q?=3F?= X-BeenThere: cerowrt-devel@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: Development issues regarding the cerowrt test router project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Mar 2015 20:29:51 -0000 I do think engineers operating networks get it, and that Comcast's engineer= s really get it, as I clarified in my followup note.=0A=0AThe issue is inde= ed prioritization of investment, engineering resources and management atten= tion. The teams at Comcast in the engineering side have been the leaders in= "bufferbloat minimizing" work, and I think they should get more recognitio= n for that.=0A=0AI disagree a little bit about not having a test that shows= the issue, and the value the test would have in demonstrating the issue to= users. Netalyzer has been doing an amazing job on this since before the b= ufferbloat term was invented. Every time I've talked about this issue I've = suggested running Netalyzer, so I have a personal set of comments from peop= le all over the world who run Netalyzer on their home networks, on hotel ne= tworks, etc.=0A=0AWhen I have brought up these measurements from Netalyzr (= which are not aimed at showing the problem as users experience) I observe a= n interesting reaction from many industry insiders: the results are not "s= exy enough for stupid users" and also "no one will care".=0A=0AI think the = reaction characterizes the problem correctly - but the second part is the m= ost serious objection. People don't need a measurement tool, they need to = know that this is why their home network sucks sometimes.=0A=0A=0A=0A=0A=0A= On Thursday, March 19, 2015 3:58pm, "Livingood, Jason" said:=0A=0A> On 3/19/15, 1:11 PM, "Dave Taht" wrote:=0A> =0A>>On Thu, Mar 19, 2015 at 6:53 AM, wrote:=0A>>> How many years has it been since Comcast said they were goin= g to fix=0A>>>bufferbloat in their network within a year?=0A> =0A> I=C2=B9m= not sure anyone ever said it=C2=B9d take a year. If someone did (even if= =0A> it=0A> was me) then it was in the days when the problem appeared less = complicated=0A> than it is and I apologize for that. Let=C2=B9s face it - t= he problem is=0A> complex and the software that has to be fixed is everywhe= re. As I said=0A> about IPv6: if it were easy, it=C2=B9d be done by now. ;-= )=0A> =0A>>>It's almost as if the cable companies don't want OTT video or= =0A>>>simultaneous FTP and interactive gaming to work. Of course not. They'= d=0A>>>never do that.=0A> =0A> Sorry, but that seems a bit unfair. It flies= in the face of what we have=0A> done and are doing. We=C2=B9ve underwritte= n some of Dave=C2=B9s work, we got=0A> CableLabs to underwrite AQM work, an= d I personally pushed like heck to get=0A> AQM built into the default D3.1 = spec (had CTO-level awareness & support,=0A> and was due to Greg White=C2= =B9s work at CableLabs). We are starting to field=0A> test D3.1 gear now, b= y the way. We made some bad bets too, such as trying=0A> to underwrite an O= penWRT-related program with ISC, but not every tactic=0A> will always be a = winner.=0A> =0A> As for existing D3.0 gear, it=C2=B9s not for lack of tryin= g. Has any DOCSIS=0A> network of any scale in the world solved it? If so, I= have something to=0A> use to learn from and apply here at Comcast - and I= =C2=B9d **love** an=0A> introduction to someone who has so I can get this i= nfo.=0A> =0A> But usually there are rational explanations for why something= is still not=0A> done. One of them is that the at-scale operational issues= are more=0A> complicated that some people realize. And there is always a c= ase of=0A> prioritization - meaning things like running out of IPv4 address= es and not=0A> having service trump more subtle things like buffer bloat (a= nd the effort=0A> to get vendors to support v6 has been tremendous).=0A> = =0A>>I do understand there are strong forces against us, especially in the = USA.=0A> =0A> I=C2=B9m not sure there are any forces against this issue. It= =C2=B9s more a=0A> question=0A> of awareness - it is not apparent it is mor= e urgent than other work in=0A> everyone=C2=B9s backlog. For example, the n= umber of ISP customers even aware of=0A> buffer bloat is probably 0.001%; i= f customers aren=C2=B9t asking for it, the=0A> product managers have a toug= h time arguing to prioritize buffer bloat work=0A> over new feature X or Y.= =0A> =0A> One suggestion I have made to increase awareness is that there be= a nice,=0A> web-based, consumer-friendly latency under load / bloat test t= hat you=0A> could get people to run as they do speed tests today. (If someo= ne thinks=0A> they can actually deliver this, I will try to fund it - ping = me off-list.)=0A> I also think a better job can be done explaining buffer b= loat - it=C2=B9s hard=0A> to make an =C5=92elevator pitch=C2=B9 about it.= =0A> =0A> It reminds me a bit of IPv6 several years ago. Rather than saying= in=0A> essence =C5=92you operators are dummies=C2=B9 for not already fixin= g this, maybe=0A> assume the engineers all =C5=92get it=C2=B9 and what to d= o it. Because we really=0A> do=0A> get it and want to do something about it= . Then ask those operators what=0A> they need to convince their leadership = and their suppliers and product=0A> managers and whomever else that it need= s to be resourced more effectively=0A> (see above for example).=0A> =0A> We= =C2=B9re at least part of the way there in DOCSIS networks. It is in D3.1 b= y=0A> default, and we=C2=B9re starting trials now. And probably within 18-2= 4 months=0A> we won=C2=B9t buy any DOCSIS CPE that is not 3.1.=0A> =0A> The= question for me is how and when to address it in DOCSIS 3.0.=0A> =0A> - Ja= son=0A> =0A> =0A> =0A> =0A