From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ob0-x22e.google.com (mail-ob0-x22e.google.com [IPv6:2607:f8b0:4003:c01::22e]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by huchra.bufferbloat.net (Postfix) with ESMTPS id CDD8221F324; Fri, 20 Mar 2015 06:48:29 -0700 (PDT) Received: by obdfc2 with SMTP id fc2so78281346obd.3; Fri, 20 Mar 2015 06:48:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=csTVjUKqptGHK34f210E3v1VhfbBsQnvovI08Er2Li0=; b=ny/Oag0DMJ8oYBcYFoID7qDVBECdUNSqeY7YorSR8MmyeI7S5SOO4sHjDV58FAf5c3 NvBy2C+ljzGCLky6WXTFwHyMPM+8gnHChT1MBIdmZoePpfviUNnPPt23OP6Q3yRJkZ4c 4XRgOPA1dAnqyrUNNs6S+GpS6IDmfFOjwGi8l8CmWTAFPmvC1IFN+KwnGEWe+5ez0hKc sDmjneGyC398srgoYWMZC4NRHLMwg7iArHWHfvaDCXPI/2wOYxJH2xQdyxDRDJYBaXci qB1J56F4KetBv54YGZf7Vvz+7ZuQn5aF0MgNQ0rhEeGn3NVghiDZeXZF1qDiuZ/xuTUC FNcA== MIME-Version: 1.0 X-Received: by 10.182.20.237 with SMTP id q13mr45574907obe.82.1426859308550; Fri, 20 Mar 2015 06:48:28 -0700 (PDT) Sender: gettysjim@gmail.com Received: by 10.76.171.230 with HTTP; Fri, 20 Mar 2015 06:48:28 -0700 (PDT) In-Reply-To: References: <20150316203532.05BD21E2@taggart.lackof.org> <123130.1426635142@turing-police.cc.vt.edu> <15A0911A-E3B7-440A-A26B-C5E1489EA98B@viagenie.ca> <1426773234.362612992@apps.rackspace.com> Date: Fri, 20 Mar 2015 09:48:28 -0400 X-Google-Sender-Auth: 3qCITzrpA5ffAU_WJtcRUAPiB28 Message-ID: From: Jim Gettys To: "Livingood, Jason" Content-Type: multipart/alternative; boundary=e89a8f838e093b11250511b8927e Cc: "cerowrt-devel@lists.bufferbloat.net" , bloat Subject: Re: [Cerowrt-devel] [Bloat] DOCSIS 3+ recommendation? X-BeenThere: cerowrt-devel@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: Development issues regarding the cerowrt test router project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Mar 2015 13:48:59 -0000 --e89a8f838e093b11250511b8927e Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On Thu, Mar 19, 2015 at 3:58 PM, Livingood, Jason < Jason_Livingood@cable.comcast.com> wrote: > On 3/19/15, 1:11 PM, "Dave Taht" wrote: > > >On Thu, Mar 19, 2015 at 6:53 AM, wrote: > >> How many years has it been since Comcast said they were going to fix > >>bufferbloat in their network within a year? > > I=C2=B9m not sure anyone ever said it=C2=B9d take a year. If someone did = (even if it > was me) then it was in the days when the problem appeared less complicate= d > than it is and I apologize for that. Let=C2=B9s face it - the problem is > complex and the software that has to be fixed is everywhere. As I said > about IPv6: if it were easy, it=C2=B9d be done by now. ;-) > =E2=80=8BI think this was the hope that the buffer size control feature in = Docsis could at least be used to cut bufferbloat down to the "traditional" 100ms level, as I remember the sequence of events. But reality intervened: buggy implementations by too many vendors, is what I remember hearing from Rich Woundy. =E2=80=8B > > >>It's almost as if the cable companies don't want OTT video or > >>simultaneous FTP and interactive gaming to work. Of course not. They'd > >>never do that. > > Sorry, but that seems a bit unfair. It flies in the face of what we have > done and are doing. We=C2=B9ve underwritten some of Dave=C2=B9s work, we = got > CableLabs to underwrite AQM work, and I personally pushed like heck to ge= t > AQM built into the default D3.1 spec (had CTO-level awareness & support, > and was due to Greg White=C2=B9s work at CableLabs). We are starting to f= ield > test D3.1 gear now, by the way. We made some bad bets too, such as trying > to underwrite an OpenWRT-related program with ISC, but not every tactic > will always be a winner. > > As for existing D3.0 gear, it=C2=B9s not for lack of trying. Has any DOCS= IS > network of any scale in the world solved it? If so, I have something to > use to learn from and apply here at Comcast - and I=C2=B9d **love** an > introduction to someone who has so I can get this info. > > But usually there are rational explanations for why something is still no= t > done. One of them is that the at-scale operational issues are more > complicated that some people realize. And there is always a case of > prioritization - meaning things like running out of IPv4 addresses and no= t > having service trump more subtle things like buffer bloat (and the effort > to get vendors to support v6 has been tremendous). > > >I do understand there are strong forces against us, especially in the US= A. > > I=C2=B9m not sure there are any forces against this issue. It=C2=B9s more= a question > of awareness - it is not apparent it is more urgent than other work in > everyone=C2=B9s backlog. For example, the number of ISP customers even aw= are of > buffer bloat is probably 0.001%; if customers aren=C2=B9t asking for it, = the > product managers have a tough time arguing to prioritize buffer bloat wor= k > over new feature X or Y. > =E2=80=8BI agree with Jason on this one. We have to take bufferbloat mains= tream to generate "market pull". I've been reluctant in the past before we had solutions in hand: very early in this quest, Dave Clark noted: =E2=80=8B"Yelling fire without having the exits marked" could be counter productive. I think we have the exits marked now. Time to yell "Fire". Even when you get to engineers in the organizations who build the equipment, it's hard. First you have to explain that "more is not better", and "some packet loss is good for you". Day to day market pressures for other features mean that: 1) many/most of the engineers =E2=80=8Bdon't see that =E2=80=8Bas what they need to do in the next quarte= r/year. 2) their management don't see that working on it should take any of their time. It won't help them sell the next set of gear. ***So we have to generate demand from the market.*** Now, I can see a couple ways to do this: 1) help expose the problem, preferably in a dead simple way that everyone sees. If we can get Ookla to add a simple test to their test system, this would be a good start. If not, other test sites are needed. Nice as Netalyzer is, it a) tops out around 20Mbps, and b) buries the buffering results among 50 other numbers. 2) Markets such as gaming are large, and very latency sensitive. Even better, lots of geeks hang out there. So investing in educating that submarket may help pull things through the system overall. 3) Competitive pressures can be very helpful: but this requires at least one significant player in each product category to "get it". So these are currently slow falling dominoes. > One suggestion I have made to increase awareness is that there be a nice, > web-based, consumer-friendly latency under load / bloat test that you > could get people to run as they do speed tests today. (If someone thinks > they can actually deliver this, I will try to fund it - ping me off-list.= ) > I also think a better job can be done explaining buffer bloat - it=C2=B9s= hard > to make an =C5=92elevator pitch=C2=B9 about it. > =E2=80=8BYeah, the elevator pitch is hard, since a number of things around bufferbloat are counter intuitive. I know, I've tried, and not really succeeded. The best kinds of metaphors have been traffic related ("building parking lots at all the bottlenecks"), and explanations like "packet loss is how the Internet enforces speed limits" http://www.circleid.com/posts/20150228_packet_loss_how_the_internet_enforce= s_speed_limits/ . =E2=80=8B > > It reminds me a bit of IPv6 several years ago. Rather than saying in > essence =C5=92you operators are dummies=C2=B9 for not already fixing this= , maybe > assume the engineers all =C5=92get it=C2=B9 and what to do it. =E2=80=8BMany/most practicing engineers are still unaware of it, or if they= have heard the word bufferbloat, still don't "get it" that they see bufferbloat's effects all the time. =E2=80=8B > Because we really do > get it and want to do something about it. Then ask those operators what > they need to convince their leadership and their suppliers and product > managers and whomever else that it needs to be resourced more effectively > (see above for example). > > We=C2=B9re at least part of the way there in DOCSIS networks. It is in D3= .1 by > default, and we=C2=B9re starting trials now. And probably within 18-24 mo= nths > we won=C2=B9t buy any DOCSIS CPE that is not 3.1. > > The question for me is how and when to address it in DOCSIS 3.0. > =E2=80=8BWe should talk at IETF. =E2=80=8B > > - Jason > > > > _______________________________________________ > Bloat mailing list > Bloat@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/bloat > --e89a8f838e093b11250511b8927e Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable

On Thu, Ma= r 19, 2015 at 3:58 PM, Livingood, Jason <Jason_Livingood@c= able.comcast.com> wrote:
On 3/19/15, 1:11 PM, "Dave Taht" <dave.taht@gmail.com> wrote:

>On Thu, Mar 19, 2015 at 6:53 AM,=C2=A0 <dpreed@reed.com> wrote:
>> How many years has it been since Comcast said they were going to f= ix
>>bufferbloat in their network within a year?

I=C2=B9m not sure anyone ever said it=C2=B9d take a year. If someone= did (even if it
was me) then it was in the days when the problem appeared less complicated<= br> than it is and I apologize for that. Let=C2=B9s face it - the problem is complex and the software that has to be fixed is everywhere. As I said
about IPv6: if it were easy, it=C2=B9d be done by now. ;-)
=

=E2=80=8BI think this was the hope that the buffer size cont= rol feature in Docsis could at least be used to cut bufferbloat down to the= "traditional" 100ms level, as I remember the sequence of events.= =C2=A0 But reality intervened: buggy implementations by too many vendors, i= s what I remember hearing from Rich Woundy.
=E2=80=8B
=C2= =A0

>>It's almost as if the cable companies don't want OTT video = or
>>simultaneous FTP and interactive gaming to work. Of course not. The= y'd
>>never do that.

Sorry, but that seems a bit unfair. It flies in the face of what we = have
done and are doing. We=C2=B9ve underwritten some of Dave=C2=B9s work, we go= t
CableLabs to underwrite AQM work, and I personally pushed like heck to get<= br> AQM built into the default D3.1 spec (had CTO-level awareness & support= ,
and was due to Greg White=C2=B9s work at CableLabs). We are starting to fie= ld
test D3.1 gear now, by the way. We made some bad bets too, such as trying to underwrite an OpenWRT-related program with ISC, but not every tactic
will always be a winner.

As for existing D3.0 gear, it=C2=B9s not for lack of trying. Has any DOCSIS=
network of any scale in the world solved it? If so, I have something to
use to learn from and apply here at Comcast - and I=C2=B9d **love** an
introduction to someone who has so I can get this info.

But usually there are rational explanations for why something is still not<= br> done. One of them is that the at-scale operational issues are more
complicated that some people realize. And there is always a case of
prioritization - meaning things like running out of IPv4 addresses and not<= br> having service trump more subtle things like buffer bloat (and the effort to get vendors to support v6 has been tremendous).

>I do understand there are strong forces against us, especially in the U= SA.

I=C2=B9m not sure there are any forces against this issue. It=C2=B9s= more a question
of awareness - it is not apparent it is more urgent than other work in
everyone=C2=B9s backlog. For example, the number of ISP customers even awar= e of
buffer bloat is probably 0.001%; if customers aren=C2=B9t asking for it, th= e
product managers have a tough time arguing to prioritize buffer bloat work<= br> over new feature X or Y.

=E2=80=8BI agree wi= th Jason on this one.=C2=A0 We have to take bufferbloat mainstream to gener= ate "market pull". I've been reluctant in the past before we = had solutions in hand: very early in this quest, Dave Clark noted:
=E2=80=8B&qu= ot;Yelling fire without having the exits marked" could be counter prod= uctive.=C2=A0 I think we have the exits marked now.=C2=A0 Time to yell &quo= t;Fire".

Even when you get to engineers in = the organizations who build the equipment, it's hard.=C2=A0 First you h= ave to explain that "more is not better", and "some packet l= oss is good for you".

Day to day market pre= ssures for other features mean that:
=C2=A0 1) many/most of the = engineers
=C2=A0
=E2=80=8Bdon't see that =E2=80=8Bas what they need to d= o in the next quarter/year.
=C2=A0 2) their management don't= see that working on it should take any of their time.=C2=A0 It won't h= elp them sell the next set of gear.

***So we hav= e to generate demand from the market.***

Now, I = can see a couple ways to do this:

1) help expose= the problem, preferably in a dead simple way that everyone sees.=C2=A0 If = we can get Ookla to add a simple test to their test system, this would be a= good start.=C2=A0 If not, other test sites are needed.=C2=A0 Nice as Netal= yzer is, it a) tops out around 20Mbps, and b) buries the buffering results = among 50 other numbers.
2) Markets such as gaming are large, an= d very latency sensitive.=C2=A0 Even better, lots of geeks hang out there.= =C2=A0 So investing in educating that submarket may help pull things throug= h the system overall.
3) Competitive pressures can be very helpf= ul: but this requires at least one significant player in each product categ= ory to "get it".=C2=A0 So these are currently slow falling domino= es.


One suggestion I have made to increase awareness is that there be a nice, web-based, consumer-friendly latency under load / bloat test that you
could get people to run as they do speed tests today. (If someone thinks they can actually deliver this, I will try to fund it - ping me off-list.)<= br> I also think a better job can be done explaining buffer bloat - it=C2=B9s h= ard
to make an =C5=92elevator pitch=C2=B9 about it.

=E2=80=8BYeah, the elevator pitch is hard, since a number of things aro= und bufferbloat are counter intuitive. I know, I've tried, and not real= ly succeeded.=C2=A0 The best kinds of metaphors have been traffic related (= "building parking lots at all the bottlenecks"), =C2=A0and explan= ations like "packet loss is how the Internet enforces speed limits&quo= t;=C2=A0http://www.circleid.com/posts/20150228_= packet_loss_how_the_internet_enforces_speed_limits/.
=E2=80= =8B
=C2=A0

It reminds me a bit of IPv6 several years ago. Rather than saying in
essence =C5=92you operators are dummies=C2=B9 for not already fixing this, = maybe
assume the engineers all =C5=92get it=C2=B9 and what to do it.
=

=E2=80=8BMany/most practicing engineers are still unaware of= it, or if they have heard the word bufferbloat, still don't "get = it" that they see bufferbloat's effects all the time.
<= div>
= =E2=80=8B
=C2=A0
Because we really do
get it and want to do something about it. Then ask those operators what
they need to convince their leadership and their suppliers and product
managers and whomever else that it needs to be resourced more effectively (see above for example).

We=C2=B9re at least part of the way there in DOCSIS networks. It is in D3.1= by
default, and we=C2=B9re starting trials now. And probably within 18-24 mont= hs
we won=C2=B9t buy any DOCSIS CPE that is not 3.1.

The question for me is how and when to address it in DOCSIS 3.0.

=E2=80=8BWe should talk at IETF.
=E2=80=8B<= /div>=C2=A0

- Jason



_______________________________________________
Bloat mailing list
Bloat@lists.bufferbloat.net<= /a>
= https://lists.bufferbloat.net/listinfo/bloat

--e89a8f838e093b11250511b8927e--