From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp181.iad.emailsrvr.com (smtp181.iad.emailsrvr.com [207.97.245.181]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by huchra.bufferbloat.net (Postfix) with ESMTPS id C062C21F0E8 for ; Sat, 2 Mar 2013 06:20:30 -0800 (PST) Received: from localhost (localhost.localdomain [127.0.0.1]) by smtp28.relay.iad1a.emailsrvr.com (SMTP Server) with ESMTP id 22EA1E06C5; Sat, 2 Mar 2013 09:20:29 -0500 (EST) X-Virus-Scanned: OK Received: from legacy18.wa-web.iad1a (legacy18.wa-web.iad1a.rsapps.net [192.168.4.108]) by smtp28.relay.iad1a.emailsrvr.com (SMTP Server) with ESMTP id E774AE045D; Sat, 2 Mar 2013 09:20:28 -0500 (EST) Received: from reed.com (localhost.localdomain [127.0.0.1]) by legacy18.wa-web.iad1a (Postfix) with ESMTP id D43A336002E; Sat, 2 Mar 2013 09:20:28 -0500 (EST) Received: by apps.rackspace.com (Authenticated sender: dpreed@reed.com, from: dpreed@reed.com) with HTTP; Sat, 2 Mar 2013 09:20:28 -0500 (EST) Date: Sat, 2 Mar 2013 09:20:28 -0500 (EST) From: dpreed@reed.com To: "Ketan Kulkarni" MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_20130302092028000000_75305" Importance: Normal X-Priority: 3 (Normal) X-Type: html In-Reply-To: References: <1362077845.101114942@apps.rackspace.com> <1362152412.065117928@apps.rackspace.com> <1362155228.00438500@apps.rackspace.com> Message-ID: <1362234028.867816955@apps.rackspace.com> X-Mailer: webmail7.0 Cc: "cerowrt-devel@lists.bufferbloat.net" , bloat Subject: Re: [Cerowrt-devel] Google working on experimental 3.8 Linux kernel for Android X-BeenThere: cerowrt-devel@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: Development issues regarding the cerowrt test router project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 02 Mar 2013 14:20:31 -0000 ------=_20130302092028000000_75305 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable =0AHi Ketan -=0A =0AIt is possible for good architects to simplify rather t= han to ramify. It takes clear understanding of the system as a whole, a un= ifying perspective, and a goal to make the system work extremely well and s= imply.=0A =0AOne of the key insights into how to do this was the choice of = features included in the IP "layer" of the Internet stack. That is - almos= t none. And if you read history, the IP layer got simpler as features (lik= e TOS) that had no sensible definition were de facto deprecated, by failure= to be utilized for any useful purpose.=0A =0AThis process works for well-a= rchitected abstractions.=0A =0AAnd it is why the original Internet team inc= luded people with expertise in radio networks, shared-medium LANs, etc., en= d-to-end cryptographic security and authentication, as well as people who u= nderstood the properties of voice conversation codecs, etc. "Features" for= those were not "omitted" - they were, instead, carefully thought through.= =0A =0ASince the goal of IP was to operate a completely technologically het= erogeneous and application heterogeneous universal network, one where new t= echnologies could be introduced without change, those were the most seriou= s issues.=0A =0AYet relative simplicity was achieved.=0A =0AThis contrasts = with the process of 3GPP and the wireless industry. Bad designs that speci= fically focus on one application (voip) without abstraction or generalizati= on, or specifically bind-in properties of one specific "bearer" technology = (circuits, scheduling algorithms), fail utterly to even fit new situations = that come up as a matter of course.=0A =0AThis is why LTE (just deployed) i= s on version 12, and IP is on version 4 (where the first 3 never were deplo= yed...), and moving to 6, which is almost exactly the same as 4.=0A =0AAnd = I claim (it's no longer a radical claim - we see the same successes in othe= r areas of architecture), that the principle of well-thought-through simpli= city is the *only* quality architectural approach.=0A =0AHowever, there are= opposing views. The latest of these is the absolute obsession with findin= g "cross layer" optimizations, and making a religion of protocol cross-laye= r "features". Which of course makes such architectures "optimal" in a sens= e, for one instant of time, one particular technology, and one Professor's = "career".=0A =0AHowever, nothing is more likely than change. And "cross la= yer" ideas essentially blow up any potential for change.=0A =0ASo it's easy= to predict that LTE is a bloody disaster in the making. It's a dysfunctio= nal system designed by a committee, driven by egos and equipment vendors' d= esire to "differentiate" merely to partition the market. (why else do no ce= llphones that support LTE work on any other operators' LTE network? That's= on purpose - it's a marketing requirement!=0A =0ASo a protocol that starte= d out to use a very nice innovation (OFDM) is now a turkey.=0A =0AIt has a = life, but not because of all those "features". In most likelihood, all tho= se "features" will eventually make it so toxic that it will be replaced qui= ckly (not by another cellular operator-centric protocol but by something qu= ite different that unifies fiber and wireless in "local" deployments, if I = were to bet).=0A =0A-----Original Message-----=0AFrom: "Ketan Kulkarni" =0ASent: Friday, March 1, 2013 10:39pm=0ATo: dpreed@reed.c= om=0ACc: "Jim Gettys" , "cerowrt-devel@lists.bufferbloa= t.net" , "bloat" =0ASubject: Re: [Cerowrt-devel] Google working on experimental 3.8 = Linux kernel for Android=0A=0A=0A=0AHi David,=0A=0AWhile I tend to agree wi= th most of the stuff, however the complexity and too many knobs in mobile n= etworks do come with the added technology.=0A=0AConsider from end-user pers= pective, getting a voice call while surfing/downloading on 2G/3G interrupts= all the download and it is annoying. =0A So when LTE provides a spec to ha= ndle voip + internet both simultaneously, its great benefit to end user.=0A= While roaming around in LTE and moving to 2G/3G network or vice-versa, the = handoff occurs seamlessly, the internet traffic is not interrupted. This wa= s not the case in previous mobile generations. End-user is more satisfied a= s it relates to the daily usage of mobile phones.=0A Similarly going ahead = we might very well have handoff from wifi to LTE - why not? =0A=0ANow for (= non-technical) mobile users, these are good and simple to have features. Bu= t from networks perspective, where and how will this complexity be handled?= definitely some nodes in the network will have to worry about LTE, UMTS, C= DMA, eHRPD and what not.=0A This gives some idea of how really complex the = network looks like -=0A[http://www.trilliumposter.com/lte.php] http://www.t= rilliumposter.com/lte.php=0A=0AFrom mobile ISP perspective, they invest hea= vy amount in getting channel license from governments. It takes years to co= ver up the amount invested. Moreover, consider e.g. facebook embedding ads = as per user's interest areas. The ads revenue benefit facebook. However, th= e mobile ISPs who carry the same data and has potentially (more relevant) i= nformation of the subscriber cannot gain from this. =0A This is a real prob= lem to solve for ISPs. Monetization is becoming more relevant to ISPs which= again will definitely lead to more complexity in network.=0A=0AFrom field = engineers, I think many of them are bound by what they are "asked" to do. T= hey have certain task to complete, with exactly what to test, and in what t= ime. Not all field engineers will think beyond that. If their senior techni= cal leads ask them to test packet drops they will test drops. Going ahead i= f they are asked to test latency they will do so with whatever available re= sources and knowledge they have. We can not expect an average engineer to t= hink beyond a certain level and do what is not expected out of them from th= eir seniors. Not all will think of the internet finally.=0A=0AEven the netw= ork vendors like Cisco, ALU or Ericsson will test the latency and all the o= ther stuff when there is real push from customers like VzW and ATT. Definit= ely the vendors can very well invest, research and come up with the latest = methods and techniques of measurement and they are doing so to some extent.= Still there is some time to come to a really good picture here.=0A=0AI th= ink the tests like RRUL is definitely a good start and is more relevant to = the ISPs VzW and ATT because its finally they own the network. If these com= panies are convinced, it will become little easy to push the vendors to do = the right stuff. It then puts the correct target for field engineers to acc= omplish. =0A We have to accept the fact that no standard till now specifies= what and how to test latencies. This is one of many of the reasons beating= bloat has been a daunting task in the complex world.=0A=0AFWIK, every ISP = has a latency and jitter budget already with every node in their network an= d these are well communicated to vendors. Now how do one measure, under wha= t scenarios is still not very clearly defined nor understood. So every one = interpret these to their knowledge and understanding and a common consensus= internally is achieved.=0A Who looks at e2e latency? Who looks at the comp= lex interactions of these nodes and their effect? - I really dont know.=0AA= dd to that the dynamic behavior of cell and radio access. All this really c= omplicates the stuff at network.=0A=0AWhile we have the right step with RRU= L, but its still a long way to go for a better picture on mobile networks.= =0A=0ASometimes "LTE =3D Long Term Employment" is an apt description.=0A=0A= Thanks,=0AKetan=0A=0A=0AOn Fri, Mar 1, 2013 at 9:57 PM, <[mailto:dpreed@re= ed.com] dpreed@reed.com> wrote:=0A=0AI don't doubt that they test. My poin= t was different - there are too many knobs and too big a parameter space to= test efectively. And that's the point.=0A =0AI realize that it's extremel= y fun to invent parameters in "standards organizations" like 3GPP. Everybo= dy has their own favorite knob, and a great rationale for some unusual, but= critically "important" customer requirement that might come up some day. = Hell, Linux has a gazillion (yes, that's a technical term in mathematics!) = parameters, almost none of which are touched. This reflects the fact that = nothing ever gets removed once added. LTE is now going into release 12, an= d it's completely ramified into "solutions" to problems that will never be = fixed in the field with those solutions. It's great for European Publicall= y Funded Academic-Industry research - lots for those "Professors" to claim = they invented.=0A =0AI've worked with telco contractors in the field. The= y don't read manuals, and they don't read specs. They have a job to do, an= d so much money to spend, and time's a wasting. They don't even work for V= erizon or ATT. They follow "specs" handed down, and charge more if you tel= l them that the specs have changed.=0A =0AThis is not how brand-new systems= get tuned.=0A =0AIt's a Clown Circus out there, and more parameters don't = help.=0A =0AThis is why "more buffering is better" continues to be the law = of the land - the spec is defined to be "no lost packets under load". I'm= sure that the primary measure under load for RRUL will be "no lost packets= " by the time it gets to field engineers in the form of "specs" - because t= hat's what they've *always* been told, and they will disregard any changes = as "typos".=0A =0AA system with more than two control parameters that inter= act in complex ways is ungovernable - and no control parameters in LTE are = "orthogonal", much less "linear" in their interaction.=0A =0A =0A =0A =0A--= ---Original Message-----=0AFrom: "Jim Gettys" <[mailto:jg@freedesktop.org] = jg@freedesktop.org>=0ASent: Friday, March 1, 2013 11:09am=0A To: "David P R= eed" <[mailto:dpreed@reed.com] dpreed@reed.com>=0A=0ACc: "Ketan Kulkarni" <= [mailto:ketkulka@gmail.com] ketkulka@gmail.com>, "[mailto:cerowrt-devel@lis= ts.bufferbloat.net] cerowrt-devel@lists.bufferbloat.net" <[mailto:cerowrt-d= evel@lists.bufferbloat.net] cerowrt-devel@lists.bufferbloat.net>=0A Subject= : Re: [Cerowrt-devel] Google working on experimental 3.8 Linux kernel for A= ndroid=0A=0A=0A=0A=0A=0A=0A=0A=0A=0A=0AOn Fri, Mar 1, 2013 at 10:40 AM, <[= mailto:dpreed@reed.com] dpreed@reed.com> wrote:=0A=0AOne wonders why all th= is complexity is necessary, and how likely it is to be "well tuned" by oper= ators and their contract installers.=0A =0AI'm willing to bet $1000 that al= l the testing that is done is "Can you hear me now" and a "speed test". No= t even something as simple and effective as RRUL.=0AActually, at least some= the the carriers do much more extensive testing; but not with the test too= ls we would like to see used (yet).=0AAn example is AT&T, where in research= , KK Ramakrishnan has a van with 20 or so laptops so he can go driving arou= nd and load up a cell in the middle of the night and get data. And he's r= esearch; the operations guys do lots of testing I gather, but more at the r= adio level.=0ANext up, is to educate KK to run RRUL.=0AAnd in my own compan= y, I've seen data, but it is too high level: e.g. performance of "web" vide= o: e.g. siverlight, flash, youtube, etc.=0AA common disease that has compli= cated all this is the propensity for companies to use Windows XP internally= for everything: since window scaling is turned off, you can't saturate a L= TE link the way you might like to do with a single TCP connection.=0A- Jim= =0A=0A=0A =0A-----Original Message-----=0AFrom: "Ketan Kulkarni" <[mailto:k= etkulka@gmail.com] ketkulka@gmail.com>=0ASent: Friday, March 1, 2013 3:00am= =0A To: "Jim Gettys" <[mailto:jg@freedesktop.org] jg@freedesktop.org>=0A Cc= : "[mailto:cerowrt-devel@lists.bufferbloat.net] cerowrt-devel@lists.bufferb= loat.net" <[mailto:cerowrt-devel@lists.bufferbloat.net] cerowrt-devel@lists= .bufferbloat.net>=0A Subject: Re: [Cerowrt-devel] Google working on experim= ental 3.8 Linux kernel for Android=0A=0A=0A=0A=0A=0AOn Fri, Mar 1, 2013 at = 1:33 AM, Jim Gettys <[mailto:jg@freedesktop.org] jg@freedesktop.org> wrote:= =0A=0AI've got a bit more insight into LTE than I did in the past, courtesy= of the last couple days.=0ATo begin with, LTE runs with several classes of= service (the call them bearers). Your VOIP traffic goes into one of them.= =0AAnd I think there is another as well that is for guaranteed bit rate tra= ffic. One transmit opportunity may have a bunch of chunks of data, and tha= t data may be destined for more than one device (IIRC). It's substantially= different than WiFi.=0AJust thought to put more light on bearer stuff:=0A= =0AThere are two ways bearers are setup: =0A1. UE initiated - where User Eq= uipment sets-up the "parameters" for bearer =0A 2. Network initiated - wher= e node like PCRF and PGW sets-up the "parameters". =0A Parameters include t= he Guaranteed bit-rates, maximum bit-rates. Something called QCI is associa= ted with bearers. The QCI parameters are authorized at PCRF (policy control= rule function) and there is certain mapping maintained at either PCRF or P= GW between QCI values and DSCP and MBRs.=0A These parameters enforcing is d= one at PGW (in such case it is termed as PCEF - policy and rule enforcement= function). So PGWs depending on bearers can certainly modify dscp bits. Th= ough these can be modified by other nodes in the network. =0A=0AThere are t= wo types of bearers: 1. Dedicated bearers - to carry traffic which need "sp= ecial" treatment 2. Default or general pupose bearers - to carry all genera= l purpose data.=0ASo generally the voip, streaming videos are passed over d= edicated bearers and apply (generally) higher GBRs, MBRs and correct dscp m= arkings.=0A And other non-latency sensitive traffic generally follows the d= efault bearer.=0A=0ATheoretical limit on maximum bearers is 11 though pract= ically most of the deployments use upto 3 bearers max.=0A=0ANote that these= parameters may very well very based on the subscriber profiles. Premium/Co= rporate subscribers can well have more GBRs and MBRs.=0A ISPs are generally= very much sensitive to the correct markings at gateways for obvious reason= s.=0A=0A=0A=0ABut most of what we think of as Internet stuff (web surfing, = dns, etc) all gets dumped into a single best effort ("BE"), class.=0AThe BE= class is definitely badly bloated; I can't say how much because I don't re= ally know yet; the test my colleague ran wasn't run long enough to be confi= dent it filled the buffers). But I will say worse than most cable modems I= 've seen. I expect this will be true to different degrees on different har= dware. The other traffic classes haven't been tested yet for bufferbloat, = though I suspect they will have it too. I was told that those classes have= much shorter queues, and when the grow, they dump the whole queues (becaus= e delivering late real time traffic is useless). But trust *and* verify...= . Verification hasn't been done for anything but BE traffic, and that hasn= 't been quantified.=0ABut each device gets a "fair" shot at bandwidth in th= e cell (or sector of a cell; they run 3 radios in each cell), where fair is= basically time based; if you are at the edge of a cell, you'll get a lot l= ess bandwidth than someone near a tower; and this fairness is guaranteed by= a scheduler than runs in the base station (called a b-nodeb, IIIRC). So t= he base station guarantees some sort of "fairness" between devices (a place= where Linux's wifi stack today fails utterly, since there is a single queu= e per device, rather than one per station).=0AWhether there are bloat probl= ems at the link level in LTE due to error correction I don't know yet; but = it wouldn't surprise me; I know there was in 3g. The people I talked to th= is morning aren't familiar with the HARQ layer in the system.=0AThe base st= ations are complicated beasts; they have both a linux system in them as wel= l as a real time operating system based device inside We don't know where = the bottle neck(s) are yet. I spent lunch upping their paranoia and gettin= g them through some conceptual hurdles (e.g. multiple bottlenecks that may = move, and the like). They will try to get me some of the data so I can hel= p them figure it out. I don't know if the data flow goes through the linux= system in the bnodeb or not, for example.=0AMost carriers are now trying t= o ensure that their backhauls from the base station are never congested, th= ough that is another known source of problems. And then there is the lack = of AQM at peering point routers.... You'd think they might run WRED there,= but many/most do not.=0A- Jim=0A=0A=0A=0A=0A=0AOn Thu, Feb 28, 2013 at 2:0= 8 PM, Dave Taht <[mailto:dave.taht@gmail.com] dave.taht@gmail.com> wrote:= =0A=0A=0A=0A=0AOn Thu, Feb 28, 2013 at 1:57 PM, <[mailto:dpreed@reed.com] = dpreed@reed.com> wrote:=0A=0ADoesn't fq_codel need an estimate of link capa= city?=0ANo, it just measures delay. Since so far as I know the outgoing por= tion of LTE is not soft-rate limited, but sensitive to the actual available= link bandwidth, fq_codel should work pretty good (if the underlying interf= aces weren't horribly overbuffired) in that direction.=0AI'm looking forwar= d to some measurements of actual buffering at the device driver/device leve= ls.=0AI don't know how inbound to the handset is managed via LTE.=0AStill q= uite a few assumptions left to smash in the above.=0A...=0Ain the home rout= er case....=0A...=0AWhen there are artificial rate limits in play (in, for = example, a cable modem/CMTS, hooked up via gigE yet rate limiting to 24up/4= mbit down), then a rate limiter (tbf,htb,hfsc) needs to be applied locally = to move that rate limiter/queue management into the local device, se we can= manage it better.=0AI'd like to be rid of the need to use htb and come up = with a rate limiter that could be adjusted dynamically from a daemon in use= rspace, probing for short all bandwidth fluctuations while monitoring the l= oad. It needent send that much data very often, to come up with a stable re= sult....=0AYou've described one soft-rate sensing scheme (piggybacking on T= CP), and I've thought up a few others, that could feed back from a daemon s= ome samples into a a soft(er) rate limiter that would keep control of the q= ueues in the home router. I am thinking it's going to take way too long to = fix the CPE and far easier to fix the home router via this method, and cert= ainly it's too painful and inaccurate to merely measure the bandwidth once,= then set a hard rate, when=0ASo far as I know the gargoyle project was exp= erimenting with this approach.=0AA problem is in places that connect more t= han one device to the cable modem... then you end up with those needing to = communicate their perception of the actual bandwidth beyond the link.=0A=0A= =0AWhere will it get that from the 4G or 3G uplink?=0A=0A=0A =0A-----Origin= al Message-----=0AFrom: "Maciej Soltysiak" <[mailto:maciej@soltysiak.com] m= aciej@soltysiak.com>=0ASent: Thursday, February 28, 2013 1:03pm=0A To: [mai= lto:cerowrt-devel@lists.bufferbloat.net] cerowrt-devel@lists.bufferbloat.ne= t=0ASubject: [Cerowrt-devel] Google working on experimental 3.8 Linux kerne= l for Android=0A=0A=0A=0AHiya,=0ALooks like Google's experimenting with 3.8= for Android: [https://android.googlesource.com/kernel/common/+/experimenta= l/android-3.8] https://android.googlesource.com/kernel/common/+/experimenta= l/android-3.8=0ASounds great if this means they will utilize fq_codel, TFO,= BQL, etc.=0AAnyway my nexus 7 says it has 3.1.10 and this 3.8 will probabl= y go to Android 5.0 so I hope Nexus 7 will get it too some day or at least = 3.3+=0APhoronix coverage: [http://www.phoronix.com/scan.php?page=3Dnews_ite= m&px=3DMTMxMzc] http://www.phoronix.com/scan.php?page=3Dnews_item&px=3DMTMx= Mzc=0ATheir 3.8 changelog: [https://android.googlesource.com/kernel/common/= +log/experimental/android-3.8] https://android.googlesource.com/kernel/comm= on/+log/experimental/android-3.8=0ARegards,=0AMaciej_______________________= ________________________=0A Cerowrt-devel mailing list=0A[mailto:Cerowrt-de= vel@lists.bufferbloat.net] Cerowrt-devel@lists.bufferbloat.net=0A[https://l= ists.bufferbloat.net/listinfo/cerowrt-devel] https://lists.bufferbloat.net/= listinfo/cerowrt-devel=0A=0A=0A=0A-- =0ADave T=C3=A4ht=0A=0AFixing bufferbl= oat with cerowrt: [http://www.teklibre.com/cerowrt/subscribe.html] http://w= ww.teklibre.com/cerowrt/subscribe.html =0A ________________________________= _______________=0A Cerowrt-devel mailing list=0A[mailto:Cerowrt-devel@lists= .bufferbloat.net] Cerowrt-devel@lists.bufferbloat.net=0A[https://lists.buff= erbloat.net/listinfo/cerowrt-devel] https://lists.bufferbloat.net/listinfo/= cerowrt-devel=0A=0A=0A_______________________________________________=0A Ce= rowrt-devel mailing list=0A[mailto:Cerowrt-devel@lists.bufferbloat.net] Cer= owrt-devel@lists.bufferbloat.net=0A[https://lists.bufferbloat.net/listinfo/= cerowrt-devel] https://lists.bufferbloat.net/listinfo/cerowrt-devel=0A=0A ------=_20130302092028000000_75305 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

= Hi Ketan -

=0A

 

=0A

It is possible for good architects to simplify rather = than to ramify.  It takes clear understanding of the system as a whole= , a unifying perspective, and a goal to make the system work extremely well= and simply.

=0A

 

=0A

One of the key insights into how to do this was th= e choice of features included in the IP "layer" of the Internet stack. = ; That is - almost none.  And if you read history, the IP layer got si= mpler as features (like TOS) that had no sensible definition were de facto = deprecated, by failure to be utilized for any useful purpose.

=0A

 

=0A

Thi= s process works for well-architected abstractions.

=0A

 

=0A

And it is why = the original Internet team included people with expertise in radio networks= , shared-medium LANs, etc., end-to-end cryptographic security and authentic= ation, as well as people who understood the properties of voice conversatio= n codecs, etc.  "Features" for those were not "omitted" - they were, i= nstead, carefully thought through.

=0A

&= nbsp;

=0A

Since the goal of IP was to op= erate a completely technologically heterogeneous and application heterogene= ous universal network, one where new technologies could be introduced witho= ut change,  those were the most serious issues.

=0A

 

=0A

Yet relative= simplicity was achieved.

=0A

 

= =0A

This contrasts with the process of 3GPP= and the wireless industry.  Bad designs that specifically focus on on= e application (voip) without abstraction or generalization, or specifically= bind-in properties of one specific "bearer" technology (circuits, scheduli= ng algorithms), fail utterly to even fit new situations that come up as a m= atter of course.

=0A

 

=0A

This is why LTE (just deployed) is on version 12= , and IP is on version 4 (where the first 3 never were deployed...), and mo= ving to 6, which is almost exactly the same as 4.

=0A

 

=0A

And I claim (it= 's no longer a radical claim - we see the same successes in other areas of = architecture), that the principle of well-thought-through simplicity is the= *only* quality architectural approach.

=0A

 

=0A

However, there are opposi= ng views.  The latest of these is the absolute obsession with finding = "cross layer" optimizations, and making a religion of protocol cross-layer = "features".  Which of course makes such architectures "optimal" in a s= ense, for one instant of time, one particular technology, and one Professor= 's "career".

=0A

 

=0A

However, nothing is more likely than change. = And "cross layer" ideas essentially blow up any potential for change.

= =0A

 

=0A

So it's easy to predict that LTE is a bloody disaster in the making.=   It's a dysfunctional system designed by a committee, driven by egos = and equipment vendors' desire to "differentiate" merely to partition the ma= rket. (why else do no cellphones that support LTE work on any other operato= rs' LTE network?  That's on purpose - it's a marketing requirement!=0A

 

=0A

So a protocol that started out to use a very nice innovation (OFDM)= is now a turkey.

=0A

 

=0A

It has a life, but not because of all those "fe= atures".  In most likelihood, all those "features" will eventually mak= e it so toxic that it will be replaced quickly (not by another cellular ope= rator-centric protocol but by something quite different that unifies fiber = and wireless in "local" deployments, if I were to bet).

=0A

 

=0A

-----Orig= inal Message-----
From: "Ketan Kulkarni" <ketkulka@gmail.com>Sent: Friday, March 1, 2013 10:39pm
To: dpreed@reed.com
Cc: "J= im Gettys" <jg@freedesktop.org>, "cerowrt-devel@lists.bufferbloat.net= " <cerowrt-devel@lists.bufferbloat.net>, "bloat" <bloat@lists.buff= erbloat.net>
Subject: Re: [Cerowrt-devel] Google working on experim= ental 3.8 Linux kernel for Android

=0A
=0A

Hi David,

While I= tend to agree with most of the stuff, however the complexity and too many = knobs in mobile networks do come with the added technology.

Cons= ider from end-user perspective, getting a voice call while surfing/download= ing on 2G/3G interrupts all the download and it is annoying.
So when= LTE provides a spec to handle voip + internet both simultaneously, its gre= at benefit to end user.
While roaming around in LTE and moving to 2G/3= G network or vice-versa, the handoff occurs seamlessly, the internet traffi= c is not interrupted. This was not the case in previous mobile generations.= End-user is more satisfied as it relates to the daily usage of mobile phon= es.
Similarly going ahead we might very well have handoff from wifi t= o LTE - why not?

Now for (non-technical) mobile users, these ar= e good and simple to have features. But from networks perspective, where an= d how will this complexity be handled? definitely some nodes in the network= will have to worry about LTE, UMTS, CDMA, eHRPD and what not.
This g= ives some idea of how really complex the network looks like -
http://www.trilliumposter.com/lt= e.php

From mobile ISP perspective, they invest heavy amount = in getting channel license from governments. It takes years to cover up the= amount invested. Moreover, consider e.g. facebook embedding ads as per use= r's interest areas. The ads revenue benefit facebook. However, the mobile I= SPs who carry the same data and has potentially (more relevant) information= of the subscriber cannot gain from this.
This is a real problem to = solve for ISPs. Monetization is becoming more relevant to ISPs which again = will definitely lead to more complexity in network.

From field e= ngineers, I think many of them are bound by what they are "asked" to do. Th= ey have certain task to complete, with exactly what to test, and in what ti= me. Not all field engineers will think beyond that. If their senior technic= al leads ask them to test packet drops they will test drops. Going ahead if= they are asked to test latency they will do so with whatever available res= ources and knowledge they have. We can not expect an average engineer to th= ink beyond a certain level and do what is not expected out of them from the= ir seniors. Not all will think of the internet finally.

Even the= network vendors like Cisco, ALU or Ericsson will test the latency and all = the other stuff when there is real push from customers like VzW and ATT. De= finitely the vendors can very well invest, research and come up with the la= test methods and techniques of measurement and they are doing so to some ex= tent.  Still there is some time to come to a really good picture here.=

I think the tests like RRUL is definitely a good start and is m= ore relevant to the ISPs VzW and ATT because its finally they own the netwo= rk. If these companies are convinced, it will become little easy to push th= e vendors to do the right stuff. It then puts the correct target for field = engineers to accomplish.
We have to accept the fact that no standard= till now specifies what and how to test latencies. This is one of many of = the reasons beating bloat has been a daunting task in the complex world.
FWIK, every ISP has a latency and jitter budget already with every= node in their network and these are well communicated to vendors. Now how = do one measure, under what scenarios is still not very clearly defined nor = understood. So every one interpret these to their knowledge and understandi= ng and a common consensus internally is achieved.
Who looks at e2e la= tency? Who looks at the complex interactions of these nodes and their effec= t? - I really dont know.
Add to that the dynamic behavior of cell and = radio access. All this really complicates the stuff at network.

= While we have the right step with RRUL, but its still a long way to go for = a better picture on mobile networks.

Sometimes "LTE =3D Long Ter= m Employment" is an apt description.

Thanks,
Ketan

=0A
On Fri, Mar 1, 2013 at 9:57 PM, <dpree= d@reed.com> wrote:
=0A
=0A

I don't doubt that they test.  My poin= t was different - there are too many knobs and too big a parameter space to= test efectively.  And that's the point.

=0A

 

=0A

I realize that it's extremely fun to invent parame= ters in "standards organizations" like 3GPP.  Everybody has their own = favorite knob, and a great rationale for some unusual, but critically "impo= rtant" customer requirement that might come up some day.  Hell, Linux = has a gazillion (yes, that's a technical term in mathematics!) parameters, = almost none of which are touched.  This reflects the fact that nothing= ever gets removed once added.  LTE is now going into release 12, and = it's completely ramified into "solutions" to problems that will never be fi= xed in the field with those solutions.  It's great for European Public= ally Funded Academic-Industry research - lots for those "Professors" to cla= im they invented.

=0A

 

=0A

I'v= e worked with telco contractors in the field.   They don't read m= anuals, and they don't read specs.  They have a job to do, and so much= money to spend, and time's a wasting.  They don't even work for Veriz= on or ATT.  They follow "specs" handed down, and charge more if you te= ll them that the specs have changed.

=0A

 

=0A

This is not how brand-new systems get tuned.

=0A

 

=0A

It's a Clown Circus out there, and = more parameters don't help.

=0A

 

=0A

This is why "more buffering is better" continues to be the law of th= e land - the spec is defined to be "no lost packets under load".  = ; I'm sure that the primary measure under load for RRUL will be "no lost pa= ckets" by the time it gets to field engineers in the form of "specs" - beca= use that's what they've *always* been told, and they will disregard any cha= nges as "typos".

=0A

 

=0A

A sy= stem with more than two control parameters that interact in complex ways is= ungovernable - and no control parameters in LTE are "orthogonal", much les= s "linear" in their interaction.

=0A

 

=0A

 

=0A

 

=0A

&n= bsp;

=0A
-----Original Message-----
From: "Jim Get= tys" <jg@freedes= ktop.org>
=0A
Sent: Friday, March 1, 2013 11:0= 9am
To: "David P Reed" <dpreed@reed.com>
=0A
=0A
Cc: "= Ketan Kulkarni" <ketkulka@gmail.com>, "cerowrt-devel@lists.bufferbloat.net" <cero= wrt-devel@lists.bufferbloat.net>
Subject: Re: [Cerowrt-devel] = Google working on experimental 3.8 Linux kernel for Android

=0A
=0A
=0A
=0A
=0A

= =0A


=0A
On = Fri, Mar 1, 2013 at 10:40 AM, <dpreed@reed.com> wrote:
= =0A
=0A

One wo= nders why all this complexity is necessary, and how likely it is to be "wel= l tuned" by operators and their contract installers.

=0A

 

=0A

I'm willing to bet $1000 that all the testi= ng that is done is "Can you hear me now" and a "speed test".  Not even= something as simple and effective as RRUL.

=0A
=0AActually, at least some the the carriers do much more extensive testing;= but not with the test tools we would like to see used (yet).
=0A
= An example is AT&T, where in research, KK Ramakrishnan has a van with 2= 0 or so laptops so he can go driving around and load up a cell in the middl= e of the night and get data.   And he's research; the operations guys = do lots of testing I gather, but more at the radio level.
=0A
Next= up, is to educate KK to run RRUL.
=0A
And in my own company, I've= seen data, but it is too high level: e.g. performance of "web" video: e.g.= siverlight, flash, youtube, etc.
=0A
A common disease that has co= mplicated all this is the propensity for companies to use Windows XP intern= ally for everything: since window scaling is turned off, you can't saturate= a LTE link the way you might like to do with a single TCP connection.=0A
- Jim
=0A
=0A
=0A
=0A

 

=0A

-----Original Message-----
From: "Ketan Kulkarn= i" <ketkulka@gma= il.com>
Sent: Friday, March 1, 2013 3:00am
To: "Jim Getty= s" <jg@freedeskt= op.org>
Cc: "cerowrt-devel@lists.bufferbloat.net" <cerowrt-= devel@lists.bufferbloat.net>
Subject: Re: [Cerowrt-devel] Goog= le working on experimental 3.8 Linux kernel for Android

=0A<= div>

=0A
On Fri, Mar 1, 2013 at 1:33 A= M, Jim Gettys <jg@freedesktop.org> wrote:
=0A=0A
I've got a bit more insight= into LTE than I did in the past, courtesy of the last couple days.=0A
= To begin with, LTE runs with several classes of service (the call them bear= ers).  Your VOIP traffic goes into one of them.
=0A
And I thi= nk there is another as well that is for guaranteed bit rate traffic.  = One transmit opportunity may have a bunch of chunks of data, and that data = may be destined for more than one device (IIRC).  It's substantially d= ifferent than WiFi.
=0A
=0A
=0A
Just thought to p= ut more light on bearer stuff:

There are two ways bearers are se= tup:
1. UE initiated - where User Equipment sets-up the "parameters" = for bearer
2. Network initiated - where node like PCRF and PGW sets-= up the "parameters".
Parameters include the Guaranteed bit-rates, ma= ximum bit-rates. Something called QCI is associated with bearers. The QCI p= arameters are authorized at PCRF (policy control rule function) and there i= s certain mapping maintained at either PCRF or PGW between QCI values and D= SCP and MBRs.
These parameters enforcing is done at PGW (in such case= it is termed as PCEF - policy and rule enforcement function). So PGWs depe= nding on bearers can certainly modify dscp bits. Though these can be modifi= ed by other nodes in the network.

There are two types of bearer= s: 1. Dedicated bearers - to carry traffic which need "special" treatment 2= . Default or general pupose bearers - to carry all general purpose data.So generally the voip, streaming videos are passed over dedicated bearer= s and apply (generally) higher GBRs, MBRs and correct dscp markings.
= And other non-latency sensitive traffic generally follows the default beare= r.

Theoretical limit on maximum bearers is 11 though practically= most of the deployments use upto 3 bearers max.

Note that these= parameters may very well very based on the subscriber profiles. Premium/Co= rporate subscribers can well have more GBRs and MBRs.
ISPs are genera= lly very much sensitive to the correct markings at gateways for obvious rea= sons.

=0A
=0A
=0A
But most of what we think of as Internet stuff (web surfin= g, dns, etc) all gets dumped into a single best effort ("BE"), class.
= =0A
The BE class is definitely badly bloated; I can't say how= much because I don't really know yet; the test my colleague ran wasn't run= long enough to be confident it filled the buffers).  But I will say w= orse than most cable modems I've seen.  I expect this will be true to = different degrees on different hardware.  The other traffic classes ha= ven't been tested yet for bufferbloat, though I suspect they will have it t= oo.  I was told that those classes have much shorter queues, and when = the grow, they dump the whole queues (because delivering late real time tra= ffic is useless).  But trust *and* verify....  Verification hasn'= t been done for anything but BE traffic, and that hasn't been quantified.=0A
But each device gets a "fair" shot at bandwidth in the cell (or= sector of a cell; they run 3 radios in each cell), where fair is basically= time based; if you are at the edge of a cell, you'll get a lot less bandwi= dth than someone near a tower; and this fairness is guaranteed by a schedul= er than runs in the base station (called a b-nodeb, IIIRC).  So the ba= se station guarantees some sort of "fairness" between devices (a place wher= e Linux's wifi stack today fails utterly, since there is a single queue per= device, rather than one per station).
=0A
Whether there are bloat= problems at the link level in LTE due to error correction I don't know yet= ; but it wouldn't surprise me; I know there was in 3g.  The people I t= alked to this morning aren't familiar with the HARQ layer in the system.=0A
The base stations are complicated beasts; they have both a linux= system in them as well as a real time operating system based device inside=  We don't know where the bottle neck(s) are yet.  I spent lunch = upping their paranoia and getting them through some conceptual hurdles (e.g= . multiple bottlenecks that may move, and the like).  They will try to= get me some of the data so I can help them figure it out.  I don't kn= ow if the data flow goes through the linux system in the bnodeb or not, for= example.
=0A
Most carriers are now trying to ensure that their ba= ckhauls from the base station are never congested, though that is another k= nown source of problems.  And then there is the lack of AQM at peering= point routers....  You'd think they might run WRED there, but many/mo= st do not.
=0A=0A
- Jim=0A
=0A
=0A
=0A


=0A
On Thu, Feb 28, 2013 at 2:08 PM, Dave = Taht <dave.taht@gmail.com> wrote:
=0A


=0A
=0A
On T= hu, Feb 28, 2013 at 1:57 PM, <dpreed@reed.com> wrote:
= =0A
=0A

Doesn'= t fq_codel need an estimate of link capacity?

=0A
=0A=
=0A
No, it just measures delay. Since so far as I know the outgoi= ng portion of LTE is not soft-rate limited, but sensitive to the actual ava= ilable link bandwidth, fq_codel should work pretty good (if the underlying = interfaces weren't horribly overbuffired) in that direction.
=0A
I= 'm looking forward to some measurements of actual buffering at the device d= river/device levels.
=0A
I don't know how inbound to the handset i= s managed via LTE.
=0A
Still quite a few assumptions left to smash= in the above.
=0A
...
=0A
in the home router case....=0A
...
=0A
When there are artificial rate limits in play (= in, for example, a cable modem/CMTS, hooked up via gigE yet rate limiting t= o 24up/4mbit down), then a rate limiter (tbf,htb,hfsc) needs to be applied = locally to move that rate limiter/queue management into the local device, s= e we can manage it better.
=0A
I'd like to be rid of the need to u= se htb and come up with a rate limiter that could be adjusted dynamically f= rom a daemon in userspace, probing for short all bandwidth fluctuations whi= le monitoring the load. It needent send that much data very often, to come = up with a stable result....
=0A
You've described one soft-rate sen= sing scheme (piggybacking on TCP), and I've thought up a few others, that c= ould feed back from a daemon some samples into a a soft(er) rate limiter th= at would keep control of the queues in the home router. I am thinking it's = going to take way too long to fix the CPE and far easier to fix the home ro= uter via this method, and certainly it's too painful and inaccurate to mere= ly measure the bandwidth once, then set a hard rate, when
=0A
So f= ar as I know the gargoyle project was experimenting with this approach.=0A
A problem is in places that connect more than one device to the c= able modem... then you end up with those needing to communicate their perce= ption of the actual bandwidth beyond the link.
=0A
=0A
=0A
=0A

Where will it= get that from the 4G or 3G uplink?

=0A
=0A
=0A

 

=0A

-----Original Message-----
From: "Macie= j Soltysiak" <= maciej@soltysiak.com>
Sent: Thursday, February 28, 2013 1:03pm<= br /> To: cerowrt-devel@lists.bufferbloat.net
Subject: [Cerowrt-deve= l] Google working on experimental 3.8 Linux kernel for Android

<= /p>=0A

=0A
Hiya,
=0A
Looks like Google's experimenting wit= h 3.8 for Android: https://android.googlesourc= e.com/kernel/common/+/experimental/android-3.8
=0A
Sounds grea= t if this means they will utilize fq_codel, TFO, BQL, etc.
=0A
Any= way my nexus 7 says it has 3.1.10 and this 3.8 will probably go to Android = 5.0 so I hope Nexus 7 will get it too some day or at least 3.3+
= =0A=0A=0A
Regards,
=0AMaciej
=0A
=0A
=0A
=0A
=0A
=0A______= _________________________________________
Cerowrt-devel mailing list<= br />Cerowrt-devel@lists.bufferbloat.net
https://lists.buffe= rbloat.net/listinfo/cerowrt-devel

=0A
=0A<= span>

--
Dave T=C3=A4ht
Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt= /subscribe.html
__________________________________= _____________
Cerowrt-devel mailing list
Cerowrt-devel@lists.buffe= rbloat.net
https://lists.bufferbloat.net/listinfo/cerowrt-= devel

=0A
=0A
=0A
=0A
=0A_______________________________________________
Cerowrt-devel mai= ling list
Cerowrt-devel@lists.bufferbloat.net
https://l= ists.bufferbloat.net/listinfo/cerowrt-devel

=0A=
=0A
=0A=0A=0A=0A=0A= =0A=0A=0A=0A=0A=0A=0A ------=_20130302092028000000_75305--