From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp152.dfw.emailsrvr.com (smtp152.dfw.emailsrvr.com [67.192.241.152]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by huchra.bufferbloat.net (Postfix) with ESMTPS id 3409421F387 for ; Fri, 20 Mar 2015 06:31:40 -0700 (PDT) Received: from smtp20.relay.dfw1a.emailsrvr.com (localhost.localdomain [127.0.0.1]) by smtp20.relay.dfw1a.emailsrvr.com (SMTP Server) with ESMTP id 571C5380081; Fri, 20 Mar 2015 09:31:39 -0400 (EDT) Received: by smtp20.relay.dfw1a.emailsrvr.com (Authenticated sender: dpreed-AT-reed.com) with ESMTPSA id 5DB293801D8; Fri, 20 Mar 2015 09:31:33 -0400 (EDT) X-Sender-Id: dpreed@reed.com Received: from [100.102.60.36] (14.sub-70-197-10.myvzw.com [70.197.10.14]) (using TLSv1.2 with cipher DHE-RSA-AES256-SHA) by 0.0.0.0:465 (trex/5.4.2); Fri, 20 Mar 2015 13:31:39 GMT User-Agent: K-@ Mail for Android X-Priority: 3 In-Reply-To: <550BD7DB.3040408@orange.com> References: <20150316203532.05BD21E2@taggart.lackof.org> <123130.1426635142@turing-police.cc.vt.edu> <15A0911A-E3B7-440A-A26B-C5E1489EA98B@viagenie.ca> <1426773234.362612992@apps.rackspace.com> <1426796961.194223197@apps.rackspace.com> <550BD7DB.3040408@orange.com> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----1W9X61IL1B4COAI1NSC2LS2C9PH3BI" Content-Transfer-Encoding: 7bit From: "David P. Reed" Date: Fri, 20 Mar 2015 06:31:29 -0700 To: MUSCARIELLO Luca IMT/OLN , Greg White , "Livingood, Jason" Message-ID: Cc: "cerowrt-devel@lists.bufferbloat.net" , bloat Subject: Re: [Bloat] [Cerowrt-devel] DOCSIS 3+ recommendation? X-BeenThere: bloat@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: General list for discussing Bufferbloat List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Mar 2015 13:32:09 -0000 ------1W9X61IL1B4COAI1NSC2LS2C9PH3BI Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable The mystery in most users' minds is that ping at a time when there is no lo= ad does tell them anything at all about why the network connection will suc= h when their kid is uploading to youtube=2E So giving them ping time is me= aningless=2E I think most network engineers think ping time is a useful me= asure of a badly bufferbloated system=2E It is not=2E The only measure is = ping time under maximum load of raw packets=2E And that requires a way to = test maximum load rtt=2E There is no problem with that =2E=2E=2E other tha= n that to understand why and how that is relevant you have to understand In= ternet congestion control=2E Having had to testify before CRTC about thi= s, I learned that most access providers (the Canadian ones) claim that such= measurements are never made as a measure of quality, and that you can calc= ulate expected latency by using Little's lemma from average throughput=2E A= nd that dropped packets are the right measure of quality of service=2E Ook= la ping time is useless in a context where even the "experts" wearing ties= from the top grossing Internet firms are so confused=2E And maybe delibera= tely misleading on purpose=2E=2E=2E they had to be forced to provide any da= ta they had about congestion in their networks by a ruling during the proc= eeding and then responded that they had no data - they never measured queue= ing delay and disputed that it mattered=2E The proper measure of congestion= was throughput=2E I kid you not=2E So Ookla ping time is useless against= such public ignorance=2E That's completely wrong for On Mar 20, 201= 5, MUSCARIELLO Luca IMT/OLN wrote: >I agr= ee=2E Having that ping included in Ookla would help a lot more > >Luca > > = >On 03/20/2015 12:18 AM, Greg White wrote: >> Netalyzr is great for network= geeks, hardly consumer-friendly, and >even so >> the "network buffer measu= rements" part is buried in 150 other >statistics=2E >> Why couldn't Ookla* = add a simultaneous "ping" test to their >throughput >> test? When was the = last time someone leaned on them? >> >> >> *I realize not everyone likes th= e Ookla tool, but it is popular and >about >> as "sexy" as you are going to= get with a network performance tool=2E >> >> -Greg >> >> >> >> On 3/19/15,= 2:29 PM, "dpreed@reed=2Ecom" wrote: >> >>> I do think = engineers operating networks get it, and that Comcast's >>> engineers reall= y get it, as I clarified in my followup note=2E >>> >>> The issue is indeed= prioritization of investment, engineering >resources >>> and management at= tention=2E The teams at Comcast in the engineering >side >>> have been the = leaders in "bufferbloat minimizing" work, and I think >they >>> should get = more recognition for that=2E >>> >>> I disagree a little bit about not havi= ng a test that shows the >issue, and >>> the value the test would have in d= emonstrating the issue to users=2E >>> Netalyzer has been doing an amazing = job on this since before the >>> bufferbloat term was invented=2E Every tim= e I've talked about this >issue >>> I've suggested running Netalyzer, so I = have a personal set of >comments >> >from people all over the world who run= Netalyzer on their home >networks, >>> on hotel networks, etc=2E >>> >>> W= hen I have brought up these measurements from Netalyzr (which are >not >>> = aimed at showing the problem as users experience) I observe an >>> interest= ing reaction from many industry insiders: the results are >not >>> "sexy e= nough for stupid users" and also "no one will care"=2E >>> >>> I think the = reaction characterizes the problem correctly - but the >second >>> part is = the most serious objection=2E People don't need a measurement >>> tool, th= ey need to know that this is why their home network sucks >>> sometimes=2E = >>> >>> >>> >>> >>> >>> On Thursday, March 19, 2015 3:58pm, "Livingood, Jas= on" >>> said: >>> >>>> On 3/19/15, = 1:11 PM, "Dave Taht" wrote: >>>> >>>>> On Thu, Ma= r 19, 2015 at 6:53 AM, wrote: >>>>>> How many years ha= s it been since Comcast said they were going to >fix >>>>>> bufferbloat in = their network within a year? >>>> I=C2=B9m not sure anyone ever said it=C2= =B9d take a year=2E If someone did >(even if >>>> it >>>> was me) then it w= as in the days when the problem appeared less >>>> complicated >>>> than it= is and I apologize for that=2E Let=C2=B9s face it - the problem is >>>> co= mplex and the software that has to be fixed is everywhere=2E As I >said >>>= > about IPv6: if it were easy, it=C2=B9d be done by now=2E ;-) >>>> >>>>>> = It's almost as if the cable companies don't want OTT video or >>>>>> simult= aneous FTP and interactive gaming to work=2E Of course not=2E >They'd >>>>>= > never do that=2E >>>> Sorry, but that seems a bit unfair=2E It flies in t= he face of what we >have >>>> done and are doing=2E We=C2=B9ve underwritten= some of Dave=C2=B9s work, we got >>>> CableLabs to underwrite AQM work, an= d I personally pushed like heck >to >>>> get >>>> AQM built into the defaul= t D3=2E1 spec (had CTO-level awareness & >support, >>>> and was due to Greg= White=C2=B9s work at CableLabs)=2E We are starting to >field >>>> test D3= =2E1 gear now, by the way=2E We made some bad bets too, such as >>>> trying= >>>> to underwrite an OpenWRT-related program with ISC, but not every >tac= tic >>>> will always be a winner=2E >>>> >>>> As for existing D3=2E0 gear, = it=C2=B9s not for lack of trying=2E Has any >DOCSIS >>>> network of any sca= le in the world solved it? If so, I have >something to >>>> use to learn fr= om and apply here at Comcast - and I=C2=B9d **love** an >>>> introduction t= o someone who has so I can get this info=2E >>>> >>>> But usually there are= rational explanations for why something is >still >>>> not >>>> done=2E On= e of them is that the at-scale operational issues are more >>>> complicated= that some people realize=2E And there is always a case of >>>> prioritizat= ion - meaning things like running out of IPv4 addresses >and >>>> not >>>> = having service trump more subtle things like buffer bloat (and the >>>> eff= ort >>>> to get vendors to support v6 has been tremendous)=2E >>>> >>>>> I = do understand there are strong forces against us, especially in >the >>>>> = USA=2E >>>> I=C2=B9m not sure there are any forces against this issue=2E It= =C2=B9s more a >>>> question >>>> of awareness - it is not apparent it is m= ore urgent than other work >in >>>> everyone=C2=B9s backlog=2E For example,= the number of ISP customers even >aware >>>> of >>>> buffer bloat is proba= bly 0=2E001%; if customers aren=C2=B9t asking for it, >the >>>> product man= agers have a tough time arguing to prioritize buffer >bloat >>>> work >>>> = over new feature X or Y=2E >>>> >>>> One suggestion I have made to increase= awareness is that there be a >>>> nice, >>>> web-based, consumer-friendly = latency under load / bloat test that >you >>>> could get people to run as t= hey do speed tests today=2E (If someone >thinks >>>> they can actually deli= ver this, I will try to fund it - ping me >>>> off-list=2E) >>>> I also thi= nk a better job can be done explaining buffer bloat - >it=C2=B9s >>>> hard = >>>> to make an =C5=92elevator pitch=C2=B9 about it=2E >>>> >>>> It reminds= me a bit of IPv6 several years ago=2E Rather than saying >in >>>> essence = =C5=92you operators are dummies=C2=B9 for not already fixing this, >maybe >= >>> assume the engineers all =C5=92get it=C2=B9 and what to do it=2E Becaus= e we >really >>>> do >>>> get it and want to do something about it=2E Then = ask those operators >what >>>> they need to convince their leadership and t= heir suppliers and >product >>>> managers and whomever else that it needs t= o be resourced more >>>> effectively >>>> (see above for example)=2E >>>> >= >>> We=C2=B9re at least part of the way there in DOCSIS networks=2E It is i= n >D3=2E1 >>>> by >>>> default, and we=C2=B9re starting trials now=2E And p= robably within 18-24 >months >>>> we won=C2=B9t buy any DOCSIS CPE that is = not 3=2E1=2E >>>> >>>> The question for me is how and when to address it in= DOCSIS 3=2E0=2E >>>> >>>> - Jason >>>> >>>> >>>> >>>> >>> -- Sent with K-= @ Mail - the evolution of emailing=2E ------1W9X61IL1B4COAI1NSC2LS2C9PH3BI Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable The mystery in most users' minds is that ping at a= time when there is no load does tell them anything at all about why the ne= twork connection will such when their kid is uploading to youtube=2E

So giving them ping time is meaningless=2E=
I think most network engineers think ping time is a us= eful measure of a badly bufferbloated system=2E It is not=2E

The only measure is ping time under maximum load o= f raw packets=2E

And that requires a= way to test maximum load rtt=2E

The= re is no problem with that =2E=2E=2E other than that to understand why and = how that is relevant you have to understand Internet congestion control=2E&= nbsp;

Having had to testify before = CRTC about this, I learned that most access providers (the Canadian ones) c= laim that such measurements are never made as a measure of quality, and tha= t you can calculate expected latency by using Little's lemma from average t= hroughput=2E And that dropped packets are the right measure of quality of s= ervice=2E

Ookla ping time  is u= seless in a context where even the "experts" wearing ties from th= e top grossing Internet firms are so confused=2E And maybe deliberately mis= leading on purpose=2E=2E=2E they had to be forced to provide any data they = had about  congestion in their networks by a ruling during the proceed= ing and then responded that they had no data - they never measured queueing= delay and disputed that it mattered=2E The proper measure of congestion wa= s throughput=2E

I kid you not=2E

So Ookla ping time is useless against s= uch public ignorance=2E



That's completely wrong for 

On Mar 20, 2015, MU= SCARIELLO Luca IMT/OLN <luca=2Emuscariello@orange=2Ecom> wrote: 
I agree=2E Having that ping included in Ookla would help a lot more

Luca

=

On 03/20/2015 12:18 AM, Greg White wrote:
Netalyzr is gre= at for network geeks, hardly consumer-friendly, and even so
the "network buffer measurements" part is buried in 150 other = statistics=2E
Why couldn't Ookla* add a simultaneous &quo= t;ping" test to their throughput
test? When was the= last time someone leaned on them?

*I realize not everyone likes the Ookla tool, but it is pop= ular and about
as "sexy" as you are going to ge= t with a network performance tool=2E

-= Greg



On 3/19/15, 2:29 PM, "dpreed@reed=2Ecom" <dpreed@reed=2Ec= om> wrote:

I do think engineers operating networks get it, an= d that Comcast's
engineers really get it, as I clarified = in my followup note=2E

The issue is in= deed prioritization of investment, engineering resources
= and management attention=2E The teams at Comcast in the engineering sidehave been the leaders in "bufferbloat minimizing"= work, and I think they
should get more recognition for t= hat=2E

I disagree a little bit about n= ot having a test that shows the issue, and
the value the = test would have in demonstrating the issue to users=2E
Ne= talyzer has been doing an amazing job on this since before the
bufferbloat term was invented=2E Every time I've talked about this is= sue
I've suggested running Netalyzer, so I have a persona= l set of comments
from people all over the world who run = Netalyzer on their home networks,
on hotel networks, etc= =2E

When I have brought up these measu= rements from Netalyzr (which are not
aimed at showing the= problem as users experience) I observe an
interesting re= action from many industry insiders: the results are not
= "sexy enough for stupid users" and also "no one will care&qu= ot;=2E

I think the reaction characteri= zes the problem correctly - but the second
part is the mo= st serious objection=2E People don't need a measurement
= tool, they need to know that this is why their home network sucks
sometimes=2E





On Thursday, M= arch 19, 2015 3:58pm, "Livingood, Jason"
<Ja= son_Livingood@cable=2Ecomcast=2Ecom> said:

On 3/19/15, 1:1= 1 PM, "Dave Taht" <dave=2Etaht@gmail=2Ecom> wrote:

On Thu, Mar 19, 2015 at 6:53 AM, <dpreed@reed=2Ecom> wrote:
How many year= s has it been since Comcast said they were going to fix
b= ufferbloat in their network within a year?
<= /blockquote>I¹m not sure anyone ever said it¹d take a year=2E If = someone did (even if
it
was me) then it= was in the days when the problem appeared less
complicat= ed
than it is and I apologize for that=2E Let¹s face= it - the problem is
complex and the software that has to= be fixed is everywhere=2E As I said
about IPv6: if it we= re easy, it¹d be done by now=2E ;-)

It's almost as if the cable companies don't want = OTT video or
simultaneous FTP and interactive gaming to w= ork=2E Of course not=2E They'd
never do that=2E
Sorry, but that seems a bit unfair=2E I= t flies in the face of what we have
done and are doing=2E= We¹ve underwritten some of Dave¹s work, we got
CableLabs to underwrite AQM work, and I personally pushed like heck to
get
AQM built into the default D3=2E1 spec= (had CTO-level awareness & support,
and was due to G= reg White¹s work at CableLabs)=2E We are starting to field
test D3=2E1 gear now, by the way=2E We made some bad bets too, such = as
trying
to underwrite an OpenWRT-rela= ted program with ISC, but not every tactic
will always be= a winner=2E

As for existing D3=2E0 ge= ar, it¹s not for lack of trying=2E Has any DOCSIS
ne= twork of any scale in the world solved it? If so, I have something to
use to learn from and apply here at Comcast - and I¹d **l= ove** an
introduction to someone who has so I can get thi= s info=2E

But usually there are ration= al explanations for why something is still
not
done=2E One of them is that the at-scale operational issues are m= ore
complicated that some people realize=2E And there is = always a case of
prioritization - meaning things like run= ning out of IPv4 addresses and
not
havi= ng service trump more subtle things like buffer bloat (and the
effort
to get vendors to support v6 has been tremen= dous)=2E

I do understand there are strong forces against us, esp= ecially in the
USA=2E
I&su= p1;m not sure there are any forces against this issue=2E It¹s more aquestion
of awareness - it is not appare= nt it is more urgent than other work in
everyone¹s b= acklog=2E For example, the number of ISP customers even aware
of
buffer bloat is probably 0=2E001%; if customers a= ren¹t asking for it, the
product managers have a tou= gh time arguing to prioritize buffer bloat
work
over new feature X or Y=2E

O= ne suggestion I have made to increase awareness is that there be a
nice,
web-based, consumer-friendly latency unde= r load / bloat test that you
could get people to run as t= hey do speed tests today=2E (If someone thinks
they can a= ctually deliver this, I will try to fund it - ping me
off= -list=2E)
I also think a better job can be done explainin= g buffer bloat - it¹s
hard
to make= an Œelevator pitch¹ about it=2E

It reminds me a bit of IPv6 several years ago=2E Rather than saying i= n
essence Œyou operators are dummies¹ for not = already fixing this, maybe
assume the engineers all &OEli= g;get it¹ and what to do it=2E Because we really
do<= br clear=3D"none">get it and want to do something about it=2E Then ask thos= e operators what
they need to convince their leadership a= nd their suppliers and product
managers and whomever else= that it needs to be resourced more
effectively
(see above for example)=2E

W= e¹re at least part of the way there in DOCSIS networks=2E It is in D3= =2E1
by
default, and we¹re startin= g trials now=2E And probably within 18-24 months
we won&s= up1;t buy any DOCSIS CPE that is not 3=2E1=2E

The question for me is how and when to address it in DOCSIS 3=2E0= =2E

- Jason







-- Sent with K-@ Mail - the evolution of emailing=2E ------1W9X61IL1B4COAI1NSC2LS2C9PH3BI--