From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp171.iad.emailsrvr.com (smtp171.iad.emailsrvr.com [207.97.245.171]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by huchra.bufferbloat.net (Postfix) with ESMTPS id C77BD21F0E8 for ; Fri, 1 Mar 2013 07:40:13 -0800 (PST) Received: from localhost (localhost.localdomain [127.0.0.1]) by smtp47.relay.iad1a.emailsrvr.com (SMTP Server) with ESMTP id 5013C3A9738; Fri, 1 Mar 2013 10:40:12 -0500 (EST) X-Virus-Scanned: OK Received: from legacy8.wa-web.iad1a (legacy8.wa-web.iad1a.rsapps.net [192.168.4.110]) by smtp47.relay.iad1a.emailsrvr.com (SMTP Server) with ESMTP id 233DC3A971E; Fri, 1 Mar 2013 10:40:12 -0500 (EST) Received: from reed.com (localhost.localdomain [127.0.0.1]) by legacy8.wa-web.iad1a (Postfix) with ESMTP id 1043F13810B; Fri, 1 Mar 2013 10:40:12 -0500 (EST) Received: by apps.rackspace.com (Authenticated sender: dpreed@reed.com, from: dpreed@reed.com) with HTTP; Fri, 1 Mar 2013 10:40:12 -0500 (EST) Date: Fri, 1 Mar 2013 10:40:12 -0500 (EST) From: dpreed@reed.com To: "Ketan Kulkarni" MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_20130301104012000000_68348" Importance: Normal X-Priority: 3 (Normal) X-Type: html In-Reply-To: References: <1362077845.101114942@apps.rackspace.com> Message-ID: <1362152412.065117928@apps.rackspace.com> X-Mailer: webmail7.0 Cc: "cerowrt-devel@lists.bufferbloat.net" Subject: Re: [Cerowrt-devel] Google working on experimental 3.8 Linux kernel for Android X-BeenThere: cerowrt-devel@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: Development issues regarding the cerowrt test router project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Mar 2013 15:40:14 -0000 ------=_20130301104012000000_68348 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable =0AOne wonders why all this complexity is necessary, and how likely it is t= o be "well tuned" by operators and their contract installers.=0A =0AI'm wil= ling to bet $1000 that all the testing that is done is "Can you hear me now= " and a "speed test". Not even something as simple and effective as RRUL.= =0A =0A-----Original Message-----=0AFrom: "Ketan Kulkarni" =0ASent: Friday, March 1, 2013 3:00am=0ATo: "Jim Gettys" =0ACc: "cerowrt-devel@lists.bufferbloat.net" =0ASubject: Re: [Cerowrt-devel] Google working on experimental= 3.8 Linux kernel for Android=0A=0A=0A=0A=0A=0AOn Fri, Mar 1, 2013 at 1:33 = AM, Jim Gettys <[mailto:jg@freedesktop.org] jg@freedesktop.org> wrote:=0A= =0AI've got a bit more insight into LTE than I did in the past, courtesy of= the last couple days.=0ATo begin with, LTE runs with several classes of se= rvice (the call them bearers). Your VOIP traffic goes into one of them.=0A= And I think there is another as well that is for guaranteed bit rate traffi= c. One transmit opportunity may have a bunch of chunks of data, and that d= ata may be destined for more than one device (IIRC). It's substantially di= fferent than WiFi.=0A=0AJust thought to put more light on bearer stuff:=0A= =0AThere are two ways bearers are setup: =0A1. UE initiated - where User Eq= uipment sets-up the "parameters" for bearer =0A 2. Network initiated - wher= e node like PCRF and PGW sets-up the "parameters". =0AParameters include th= e Guaranteed bit-rates, maximum bit-rates. Something called QCI is associat= ed with bearers. The QCI parameters are authorized at PCRF (policy control = rule function) and there is certain mapping maintained at either PCRF or PG= W between QCI values and DSCP and MBRs.=0A These parameters enforcing is do= ne at PGW (in such case it is termed as PCEF - policy and rule enforcement = function). So PGWs depending on bearers can certainly modify dscp bits. Tho= ugh these can be modified by other nodes in the network. =0A=0AThere are tw= o types of bearers: 1. Dedicated bearers - to carry traffic which need "spe= cial" treatment 2. Default or general pupose bearers - to carry all general= purpose data.=0ASo generally the voip, streaming videos are passed over de= dicated bearers and apply (generally) higher GBRs, MBRs and correct dscp ma= rkings.=0A And other non-latency sensitive traffic generally follows the de= fault bearer.=0A=0ATheoretical limit on maximum bearers is 11 though practi= cally most of the deployments use upto 3 bearers max.=0A=0ANote that these = parameters may very well very based on the subscriber profiles. Premium/Cor= porate subscribers can well have more GBRs and MBRs.=0A ISPs are generally = very much sensitive to the correct markings at gateways for obvious reasons= .=0A=0A=0A=0ABut most of what we think of as Internet stuff (web surfing, d= ns, etc) all gets dumped into a single best effort ("BE"), class.=0AThe BE = class is definitely badly bloated; I can't say how much because I don't rea= lly know yet; the test my colleague ran wasn't run long enough to be confid= ent it filled the buffers). But I will say worse than most cable modems I'= ve seen. I expect this will be true to different degrees on different hard= ware. The other traffic classes haven't been tested yet for bufferbloat, t= hough I suspect they will have it too. I was told that those classes have = much shorter queues, and when the grow, they dump the whole queues (because= delivering late real time traffic is useless). But trust *and* verify....= Verification hasn't been done for anything but BE traffic, and that hasn'= t been quantified.=0ABut each device gets a "fair" shot at bandwidth in the= cell (or sector of a cell; they run 3 radios in each cell), where fair is = basically time based; if you are at the edge of a cell, you'll get a lot le= ss bandwidth than someone near a tower; and this fairness is guaranteed by = a scheduler than runs in the base station (called a b-nodeb, IIIRC). So th= e base station guarantees some sort of "fairness" between devices (a place = where Linux's wifi stack today fails utterly, since there is a single queue= per device, rather than one per station).=0AWhether there are bloat proble= ms at the link level in LTE due to error correction I don't know yet; but i= t wouldn't surprise me; I know there was in 3g. The people I talked to thi= s morning aren't familiar with the HARQ layer in the system.=0AThe base sta= tions are complicated beasts; they have both a linux system in them as well= as a real time operating system based device inside We don't know where t= he bottle neck(s) are yet. I spent lunch upping their paranoia and getting= them through some conceptual hurdles (e.g. multiple bottlenecks that may m= ove, and the like). They will try to get me some of the data so I can help= them figure it out. I don't know if the data flow goes through the linux = system in the bnodeb or not, for example.=0AMost carriers are now trying to= ensure that their backhauls from the base station are never congested, tho= ugh that is another known source of problems. And then there is the lack o= f AQM at peering point routers.... You'd think they might run WRED there, = but many/most do not.=0A- Jim=0A=0A=0A=0A=0A=0AOn Thu, Feb 28, 2013 at 2:08= PM, Dave Taht <[mailto:dave.taht@gmail.com] dave.taht@gmail.com> wrote:=0A= =0A=0A=0A=0AOn Thu, Feb 28, 2013 at 1:57 PM, <[mailto:dpreed@reed.com] dpr= eed@reed.com> wrote:=0A=0ADoesn't fq_codel need an estimate of link capacit= y?=0ANo, it just measures delay. Since so far as I know the outgoing portio= n of LTE is not soft-rate limited, but sensitive to the actual available li= nk bandwidth, fq_codel should work pretty good (if the underlying interface= s weren't horribly overbuffired) in that direction.=0AI'm looking forward t= o some measurements of actual buffering at the device driver/device levels.= =0AI don't know how inbound to the handset is managed via LTE.=0A=0AStill q= uite a few assumptions left to smash in the above.=0A...=0Ain the home rout= er case....=0A...=0AWhen there are artificial rate limits in play (in, for = example, a cable modem/CMTS, hooked up via gigE yet rate limiting to 24up/4= mbit down), then a rate limiter (tbf,htb,hfsc) needs to be applied locally = to move that rate limiter/queue management into the local device, se we can= manage it better.=0AI'd like to be rid of the need to use htb and come up = with a rate limiter that could be adjusted dynamically from a daemon in use= rspace, probing for short all bandwidth fluctuations while monitoring the l= oad. It needent send that much data very often, to come up with a stable re= sult....=0AYou've described one soft-rate sensing scheme (piggybacking on T= CP), and I've thought up a few others, that could feed back from a daemon s= ome samples into a a soft(er) rate limiter that would keep control of the q= ueues in the home router. I am thinking it's going to take way too long to = fix the CPE and far easier to fix the home router via this method, and cert= ainly it's too painful and inaccurate to merely measure the bandwidth once,= then set a hard rate, when=0ASo far as I know the gargoyle project was exp= erimenting with this approach.=0A=0AA problem is in places that connect mor= e than one device to the cable modem... then you end up with those needing = to communicate their perception of the actual bandwidth beyond the link.=0A= =0A=0AWhere will it get that from the 4G or 3G uplink?=0A=0A=0A =0A-----Ori= ginal Message-----=0AFrom: "Maciej Soltysiak" <[mailto:maciej@soltysiak.com= ] maciej@soltysiak.com>=0ASent: Thursday, February 28, 2013 1:03pm=0A To: [= mailto:cerowrt-devel@lists.bufferbloat.net] cerowrt-devel@lists.bufferbloat= .net=0ASubject: [Cerowrt-devel] Google working on experimental 3.8 Linux ke= rnel for Android=0A=0A=0A=0AHiya,=0ALooks like Google's experimenting with = 3.8 for Android: [https://android.googlesource.com/kernel/common/+/experime= ntal/android-3.8] https://android.googlesource.com/kernel/common/+/experime= ntal/android-3.8=0ASounds great if this means they will utilize fq_codel, T= FO, BQL, etc.=0AAnyway my nexus 7 says it has 3.1.10 and this 3.8 will prob= ably go to Android 5.0 so I hope Nexus 7 will get it too some day or at lea= st 3.3+=0APhoronix coverage: [http://www.phoronix.com/scan.php?page=3Dnews_= item&px=3DMTMxMzc] http://www.phoronix.com/scan.php?page=3Dnews_item&px=3DM= TMxMzc=0ATheir 3.8 changelog: [https://android.googlesource.com/kernel/comm= on/+log/experimental/android-3.8] https://android.googlesource.com/kernel/c= ommon/+log/experimental/android-3.8=0ARegards,=0AMaciej____________________= ___________________________=0A Cerowrt-devel mailing list=0A[mailto:Cerowrt= -devel@lists.bufferbloat.net] Cerowrt-devel@lists.bufferbloat.net=0A[https:= //lists.bufferbloat.net/listinfo/cerowrt-devel] https://lists.bufferbloat.n= et/listinfo/cerowrt-devel=0A=0A=0A=0A-- =0ADave T=C3=A4ht=0A=0AFixing buffe= rbloat with cerowrt: [http://www.teklibre.com/cerowrt/subscribe.html] http:= //www.teklibre.com/cerowrt/subscribe.html =0A______________________________= _________________=0A Cerowrt-devel mailing list=0A[mailto:Cerowrt-devel@lis= ts.bufferbloat.net] Cerowrt-devel@lists.bufferbloat.net=0A[https://lists.bu= fferbloat.net/listinfo/cerowrt-devel] https://lists.bufferbloat.net/listinf= o/cerowrt-devel=0A=0A=0A_______________________________________________=0A = Cerowrt-devel mailing list=0A[mailto:Cerowrt-devel@lists.bufferbloat.net] C= erowrt-devel@lists.bufferbloat.net=0A[https://lists.bufferbloat.net/listinf= o/cerowrt-devel] https://lists.bufferbloat.net/listinfo/cerowrt-devel=0A=0A ------=_20130301104012000000_68348 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

= One wonders why all this complexity is necessary, and how likely it is to b= e "well tuned" by operators and their contract installers.

=0A

 

=0A

I'm = willing to bet $1000 that all the testing that is done is "Can you hear me = now" and a "speed test".  Not even something as simple and effective a= s RRUL.

=0A

 

=0A

-----Original Message-----
From: "Ketan Kulkarni" &l= t;ketkulka@gmail.com>
Sent: Friday, March 1, 2013 3:00am
To: "= Jim Gettys" <jg@freedesktop.org>
Cc: "cerowrt-devel@lists.buffer= bloat.net" <cerowrt-devel@lists.bufferbloat.net>
Subject: Re: [C= erowrt-devel] Google working on experimental 3.8 Linux kernel for Android

=0A


=0A
On Fri, Mar 1, 2013 at 1:33 AM, Jim Gettys <jg@freedeskt= op.org> wrote:
=0A
= =0A
I've got a bit more insight into LTE than I did in the = past, courtesy of the last couple days.=0A
To begin with, LTE runs with= several classes of service (the call them bearers).  Your VOIP traffi= c goes into one of them.
=0A
And I think there is another as well = that is for guaranteed bit rate traffic.  One transmit opportunity may= have a bunch of chunks of data, and that data may be destined for more tha= n one device (IIRC).  It's substantially different than WiFi.
=0A=
=0A
=0A
=0A
Just thought to put more light = on bearer stuff:

There are two ways bearers are setup:
1. = UE initiated - where User Equipment sets-up the "parameters" for bearer 2. Network initiated - where node like PCRF and PGW sets-up the "parame= ters".
Parameters include the Guaranteed bit-rates, maximum bit-rates= . Something called QCI is associated with bearers. The QCI parameters are a= uthorized at PCRF (policy control rule function) and there is certain mappi= ng maintained at either PCRF or PGW between QCI values and DSCP and MBRs. These parameters enforcing is done at PGW (in such case it is termed a= s PCEF - policy and rule enforcement function). So PGWs depending on bearer= s can certainly modify dscp bits. Though these can be modified by other nod= es in the network.

There are two types of bearers: 1. Dedicated= bearers - to carry traffic which need "special" treatment 2. Default or ge= neral pupose bearers - to carry all general purpose data.
So generally= the voip, streaming videos are passed over dedicated bearers and apply (ge= nerally) higher GBRs, MBRs and correct dscp markings.
And other non-l= atency sensitive traffic generally follows the default bearer.

T= heoretical limit on maximum bearers is 11 though practically most of the de= ployments use upto 3 bearers max.

Note that these parameters may= very well very based on the subscriber profiles. Premium/Corporate subscri= bers can well have more GBRs and MBRs.
ISPs are generally very much s= ensitive to the correct markings at gateways for obvious reasons.

=0A
=0A
=0ABut most of what we think of as Internet stuff (web surfing, dns, etc) all= gets dumped into a single best effort ("BE"), class.
=0A
The BE c= lass is definitely badly bloated; I can't say how much because I = don't really know yet; the test my colleague ran wasn't run long enough to = be confident it filled the buffers).  But I will say worse than most c= able modems I've seen.  I expect this will be true to different degree= s on different hardware.  The other traffic classes haven't been teste= d yet for bufferbloat, though I suspect they will have it too.  I was = told that those classes have much shorter queues, and when the grow, they d= ump the whole queues (because delivering late real time traffic is useless)= .  But trust *and* verify....  Verification hasn't been done for = anything but BE traffic, and that hasn't been quantified.
=0A
But = each device gets a "fair" shot at bandwidth in the cell (or sector of a cel= l; they run 3 radios in each cell), where fair is basically time based; if = you are at the edge of a cell, you'll get a lot less bandwidth than someone= near a tower; and this fairness is guaranteed by a scheduler than runs in = the base station (called a b-nodeb, IIIRC).  So the base station guara= ntees some sort of "fairness" between devices (a place where Linux's wifi s= tack today fails utterly, since there is a single queue per device, rather = than one per station).
=0A
Whether there are bloat problems at the= link level in LTE due to error correction I don't know yet; but it wouldn'= t surprise me; I know there was in 3g.  The people I talked to this mo= rning aren't familiar with the HARQ layer in the system.
=0A
The b= ase stations are complicated beasts; they have both a linux system in them = as well as a real time operating system based device inside  We don't = know where the bottle neck(s) are yet.  I spent lunch upping their par= anoia and getting them through some conceptual hurdles (e.g. multiple bottl= enecks that may move, and the like).  They will try to get me some of = the data so I can help them figure it out.  I don't know if the data f= low goes through the linux system in the bnodeb or not, for example.
= =0A
Most carriers are now trying to ensure that their backhauls from th= e base station are never congested, though that is another known source of = problems.  And then there is the lack of AQM at peering point routers.= ...  You'd think they might run WRED there, but many/most do not.=0A=0A
- Jim=0A
=0A
=0A
=0A=


=0A
On Thu= , Feb 28, 2013 at 2:08 PM, Dave Taht <dave.taht@gmail.com>= wrote:
=0A


=0A
=0A
On Thu, Feb 28, 2013 at 1:57 PM, <dpreed@reed.c= om> wrote:
=0A
=0A

Doesn't fq_codel need an estimate of link capacity?=

=0A
=0A
=0A
No, it just measures delay. Sin= ce so far as I know the outgoing portion of LTE is not soft-rate limited, b= ut sensitive to the actual available link bandwidth, fq_codel should work p= retty good (if the underlying interfaces weren't horribly overbuffired) in = that direction.
=0A
I'm looking forward to some measurements of ac= tual buffering at the device driver/device levels.
=0A
I don't kno= w how inbound to the handset is managed via LTE.
=0A
=0AStill quite a few assumptions left to smash in the above.
=0A
...=
=0A
in the home router case....
=0A
...
=0A
Whe= n there are artificial rate limits in play (in, for example, a cable modem/= CMTS, hooked up via gigE yet rate limiting to 24up/4mbit down), then a rate= limiter (tbf,htb,hfsc) needs to be applied locally to move that rate limit= er/queue management into the local device, se we can manage it better.=0A
I'd like to be rid of the need to use htb and come up with a rate = limiter that could be adjusted dynamically from a daemon in userspace, prob= ing for short all bandwidth fluctuations while monitoring the load. It need= ent send that much data very often, to come up with a stable result....=0A
You've described one soft-rate sensing scheme (piggybacking on TC= P), and I've thought up a few others, that could feed back from a daemon so= me samples into a a soft(er) rate limiter that would keep control of the qu= eues in the home router. I am thinking it's going to take way too long to f= ix the CPE and far easier to fix the home router via this method, and certa= inly it's too painful and inaccurate to merely measure the bandwidth once, = then set a hard rate, when
=0A
So far as I know the gargoyle proje= ct was experimenting with this approach.
=0A
=0A
A probl= em is in places that connect more than one device to the cable modem... the= n you end up with those needing to communicate their perception of the actu= al bandwidth beyond the link.
=0A
= =0A
=0A
=0A

Where will it get that from = the 4G or 3G uplink?

=0A
=0A
=0A

 

=0A

-----Original Message-----
From: "Maciej Soltysiak" &l= t;maciej@soltysia= k.com>
Sent: Thursday, February 28, 2013 1:03pm
To: cerowrt= -devel@lists.bufferbloat.net
Subject: [Cerowrt-devel] Google worki= ng on experimental 3.8 Linux kernel for Android

=0A
=0A<= div>Hiya,
=0A
Looks like Google's experimenting with 3.8 for Andro= id: https://android.googlesource.com/kernel/co= mmon/+/experimental/android-3.8
=0A
Sounds great if this means= they will utilize fq_codel, TFO, BQL, etc.
=0A
Anyway my nexus 7 = says it has 3.1.10 and this 3.8 will probably go to Android 5.0 so I hope N= exus 7 will get it too some day or at least 3.3+
=0A=0A=0A
Regards,
=0A
Maciej
= =0A
=0A
=0A
=0A
=0A
=0A____________________= ___________________________
Cerowrt-devel mailing list
Cerowrt-d= evel@lists.bufferbloat.net
https://lists.bufferbloat.net/l= istinfo/cerowrt-devel

=0A
=0A

--
Dave T=C3=A4ht

Fix= ing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.h= tml
_______________________________________________=
Cerowrt-devel mailing list
Cerowrt-devel@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cerowrt-devel

=0A
=0A
=0A=0A=0A
_________= ______________________________________
Cerowrt-devel mailing list
Cerowrt-devel@list= s.bufferbloat.net
https://lists.bufferbloat.net/listinfo/c= erowrt-devel

=0A=0A ------=_20130301104012000000_68348--