From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ve0-f176.google.com (mail-ve0-f176.google.com [209.85.128.176]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority" (verified OK)) by huchra.bufferbloat.net (Postfix) with ESMTPS id F0F0921F0E8 for ; Fri, 1 Mar 2013 00:00:01 -0800 (PST) Received: by mail-ve0-f176.google.com with SMTP id cz10so2622349veb.35 for ; Fri, 01 Mar 2013 00:00:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=RFigQVRvXq292W7zHsyhxjLuWMgQVC/BIJu6y06go6s=; b=HNs4kwd1TKY7mqrTPO9O94QFI9N0j5EtpA/PFjfJh+J4n0/mZzf65pN1/DBw8brQqX s/wkXDGIvhjtyiIbAaGBa3FQ7HAu6JR96sKeij7IFgbJMTKWy3oZShrYodDRawPfHshz OUzE1bIngHTkL4NHB9HCRsrdKawpn3qH3on+rkaPiIsordkL+b8Efu4JCSGA3WNESvWz KYCyt7RoYQyJYKcribTET6jhNqi0OPAK87jIbOUcYgNhM/KWLZ8QOWzE7CUy0gxbS3qp s4C2y2SPb4ZZFbc6iMZBi4BOzFhVEZ+gCK9kW11km13Q5J+yCPGerhBjckrzj1R9YNX/ 03OA== MIME-Version: 1.0 X-Received: by 10.52.36.234 with SMTP id t10mr3237172vdj.95.1362124800473; Fri, 01 Mar 2013 00:00:00 -0800 (PST) Received: by 10.58.106.42 with HTTP; Fri, 1 Mar 2013 00:00:00 -0800 (PST) In-Reply-To: References: <1362077845.101114942@apps.rackspace.com> Date: Fri, 1 Mar 2013 13:30:00 +0530 Message-ID: From: Ketan Kulkarni To: Jim Gettys Content-Type: multipart/alternative; boundary=20cf307813fcdf43ac04d6d86439 Cc: "cerowrt-devel@lists.bufferbloat.net" Subject: Re: [Cerowrt-devel] Google working on experimental 3.8 Linux kernel for Android X-BeenThere: cerowrt-devel@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: Development issues regarding the cerowrt test router project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Mar 2013 08:00:02 -0000 --20cf307813fcdf43ac04d6d86439 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable On Fri, Mar 1, 2013 at 1:33 AM, Jim Gettys wrote: > I've got a bit more insight into LTE than I did in the past, courtesy of > the last couple days. > > To begin with, LTE runs with several classes of service (the call them > bearers). Your VOIP traffic goes into one of them. > And I think there is another as well that is for guaranteed bit rate > traffic. One transmit opportunity may have a bunch of chunks of data, an= d > that data may be destined for more than one device (IIRC). It's > substantially different than WiFi. > Just thought to put more light on bearer stuff: There are two ways bearers are setup: 1. UE initiated - where User Equipment sets-up the "parameters" for bearer 2. Network initiated - where node like PCRF and PGW sets-up the "parameters". Parameters include the Guaranteed bit-rates, maximum bit-rates. Something called QCI is associated with bearers. The QCI parameters are authorized at PCRF (policy control rule function) and there is certain mapping maintained at either PCRF or PGW between QCI values and DSCP and MBRs. These parameters enforcing is done at PGW (in such case it is termed as PCEF - policy and rule enforcement function). So PGWs depending on bearers can certainly modify dscp bits. Though these can be modified by other nodes in the network. There are two types of bearers: 1. Dedicated bearers - to carry traffic which need "special" treatment 2. Default or general pupose bearers - to carry all general purpose data. So generally the voip, streaming videos are passed over dedicated bearers and apply (generally) higher GBRs, MBRs and correct dscp markings. And other non-latency sensitive traffic generally follows the default bearer. Theoretical limit on maximum bearers is 11 though practically most of the deployments use upto 3 bearers max. Note that these parameters may very well very based on the subscriber profiles. Premium/Corporate subscribers can well have more GBRs and MBRs. ISPs are generally very much sensitive to the correct markings at gateways for obvious reasons. > But most of what we think of as Internet stuff (web surfing, dns, etc) al= l > gets dumped into a single best effort ("BE"), class. > > The BE class is definitely badly bloated; I can't say how much because I > don't really know yet; the test my colleague ran wasn't run long enough t= o > be confident it filled the buffers). But I will say worse than most cabl= e > modems I've seen. I expect this will be true to different degrees on > different hardware. The other traffic classes haven't been tested yet fo= r > bufferbloat, though I suspect they will have it too. I was told that tho= se > classes have much shorter queues, and when the grow, they dump the whole > queues (because delivering late real time traffic is useless). But trust > *and* verify.... Verification hasn't been done for anything but BE > traffic, and that hasn't been quantified. > > But each device gets a "fair" shot at bandwidth in the cell (or sector of > a cell; they run 3 radios in each cell), where fair is basically time > based; if you are at the edge of a cell, you'll get a lot less bandwidth > than someone near a tower; and this fairness is guaranteed by a scheduler > than runs in the base station (called a b-nodeb, IIIRC). So the base > station guarantees some sort of "fairness" between devices (a place where > Linux's wifi stack today fails utterly, since there is a single queue per > device, rather than one per station). > > Whether there are bloat problems at the link level in LTE due to error > correction I don't know yet; but it wouldn't surprise me; I know there wa= s > in 3g. The people I talked to this morning aren't familiar with the HARQ > layer in the system. > > The base stations are complicated beasts; they have both a linux system i= n > them as well as a real time operating system based device inside We don'= t > know where the bottle neck(s) are yet. I spent lunch upping their parano= ia > and getting them through some conceptual hurdles (e.g. multiple bottlenec= ks > that may move, and the like). They will try to get me some of the data s= o > I can help them figure it out. I don't know if the data flow goes throug= h > the linux system in the bnodeb or not, for example. > > Most carriers are now trying to ensure that their backhauls from the base > station are never congested, though that is another known source of > problems. And then there is the lack of AQM at peering point routers.... > You'd think they might run WRED there, but many/most do not. > - Jim > > > > > > On Thu, Feb 28, 2013 at 2:08 PM, Dave Taht wrote: > >> >> >> On Thu, Feb 28, 2013 at 1:57 PM, wrote: >> >>> Doesn't fq_codel need an estimate of link capacity? >>> >> >> No, it just measures delay. Since so far as I know the outgoing portion >> of LTE is not soft-rate limited, but sensitive to the actual available l= ink >> bandwidth, fq_codel should work pretty good (if the underlying interface= s >> weren't horribly overbuffired) in that direction. >> >> I'm looking forward to some measurements of actual buffering at the >> device driver/device levels. >> >> I don't know how inbound to the handset is managed via LTE. >> >> Still quite a few assumptions left to smash in the above. >> >> ... >> >> in the home router case.... >> >> ... >> >> When there are artificial rate limits in play (in, for example, a cable >> modem/CMTS, hooked up via gigE yet rate limiting to 24up/4mbit down), th= en >> a rate limiter (tbf,htb,hfsc) needs to be applied locally to move that r= ate >> limiter/queue management into the local device, se we can manage it bett= er. >> >> I'd like to be rid of the need to use htb and come up with a rate limite= r >> that could be adjusted dynamically from a daemon in userspace, probing f= or >> short all bandwidth fluctuations while monitoring the load. It needent s= end >> that much data very often, to come up with a stable result.... >> >> You've described one soft-rate sensing scheme (piggybacking on TCP), and >> I've thought up a few others, that could feed back from a daemon some >> samples into a a soft(er) rate limiter that would keep control of the >> queues in the home router. I am thinking it's going to take way too long= to >> fix the CPE and far easier to fix the home router via this method, and >> certainly it's too painful and inaccurate to merely measure the bandwidt= h >> once, then set a hard rate, when >> >> So far as I know the gargoyle project was experimenting with this >> approach. >> >> A problem is in places that connect more than one device to the cable >> modem... then you end up with those needing to communicate their percept= ion >> of the actual bandwidth beyond the link. >> >> Where will it get that from the 4G or 3G uplink? >>> >>> >>> >>> -----Original Message----- >>> From: "Maciej Soltysiak" >>> Sent: Thursday, February 28, 2013 1:03pm >>> To: cerowrt-devel@lists.bufferbloat.net >>> Subject: [Cerowrt-devel] Google working on experimental 3.8 Linux kerne= l >>> for Android >>> >>> Hiya, >>> Looks like Google's experimenting with 3.8 for Android: >>> https://android.googlesource.com/kernel/common/+/experimental/android-3= .8 >>> Sounds great if this means they will utilize fq_codel, TFO, BQL, etc. >>> Anyway my nexus 7 says it has 3.1.10 and this 3.8 will probably go to >>> Android 5.0 so I hope Nexus 7 will get it too some day or at least 3.3+ >>> Phoronix coverage: >>> http://www.phoronix.com/scan.php?page=3Dnews_item&px=3DMTMxMzc >>> Their 3.8 changelog: >>> https://android.googlesource.com/kernel/common/+log/experimental/androi= d-3.8 >>> Regards, >>> Maciej >>> >>> _______________________________________________ >>> Cerowrt-devel mailing list >>> Cerowrt-devel@lists.bufferbloat.net >>> https://lists.bufferbloat.net/listinfo/cerowrt-devel >>> >>> >> >> >> -- >> Dave T=E4ht >> >> Fixing bufferbloat with cerowrt: >> http://www.teklibre.com/cerowrt/subscribe.html >> _______________________________________________ >> Cerowrt-devel mailing list >> Cerowrt-devel@lists.bufferbloat.net >> https://lists.bufferbloat.net/listinfo/cerowrt-devel >> >> > > _______________________________________________ > Cerowrt-devel mailing list > Cerowrt-devel@lists.bufferbloat.net > https://lists.bufferbloat.net/listinfo/cerowrt-devel > > --20cf307813fcdf43ac04d6d86439 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable

On Fri, Mar 1, 2013 at 1:33 AM, Jim Gett= ys <jg@freedesktop.org> wrote:
I've got a bit more insight into LTE than I did in the= past, courtesy of the last couple days.

To begin with, = LTE runs with several classes of service (the call them bearers). =A0Your V= OIP traffic goes into one of them.
And I think there is another as well that is for guaranteed bit rate t= raffic. =A0One transmit opportunity may have a bunch of chunks of data, and= that data may be destined for more than one device (IIRC). =A0It's sub= stantially different than WiFi.
=A0
Just thought to put more light on bea= rer stuff:

There are two ways bearers are setup:
1. UE initiated= - where User Equipment sets-up the "parameters" for bearer
2. Network initiated - where node like PCRF and PGW sets-up the "param= eters".
Parameters include the Guaranteed bit-rates, maximum bit-r= ates. Something called QCI is associated with bearers. The QCI parameters a= re authorized at PCRF (policy control rule function) and there is certain m= apping maintained at either PCRF or PGW between QCI values and DSCP and MBR= s.
These parameters enforcing is done at PGW (in such case it is termed as PCE= F - policy and rule enforcement function). So PGWs depending on bearers can= certainly modify dscp bits. Though these can be modified by other nodes in= the network.

There are two types of bearers: 1. Dedicated bearers - to carry traffic= which need "special" treatment 2. Default or general pupose bear= ers - to carry all general purpose data.
So generally the voip, streamin= g videos are passed over dedicated bearers and apply (generally) higher GBR= s, MBRs and correct dscp markings.
And other non-latency sensitive traffic generally follows the default beare= r.

Theoretical limit on maximum bearers is 11 though practically mos= t of the deployments use upto 3 bearers max.

Note that these paramet= ers may very well very based on the subscriber profiles. Premium/Corporate = subscribers can well have more GBRs and MBRs.
ISPs are generally very much sensitive to the correct markings at gateways = for obvious reasons.


But most of what we think of as Internet stuff (web sur= fing, dns, etc) all gets dumped into a single best effort ("BE"),= class.

The BE class is=A0definitely=A0badly bloat= ed; I can't say how much because I don't really know yet; the test = my colleague ran wasn't run long enough to be confident it filled the b= uffers). =A0But I will say worse than most cable modems I've seen. =A0I= expect this will be true to different degrees on different hardware. =A0Th= e other traffic classes haven't been tested yet for bufferbloat, though= I suspect they will have it too. =A0I was told that those classes have muc= h shorter queues, and when the grow, they dump the whole queues (because de= livering late real time traffic is useless). =A0But trust *and* verify.... = =A0Verification hasn't been done for anything but BE traffic, and that = hasn't been quantified.

But each device gets a "fair" shot at bandwid= th in the cell (or sector of a cell; they run 3 radios in each cell), where= fair is basically time based; if you are at the edge of a cell, you'll= get a lot less bandwidth than someone near a tower; and this fairness is g= uaranteed by a scheduler than runs in the base station (called a b-nodeb, I= IIRC). =A0So the base station guarantees some sort of "fairness" = between devices (a place where Linux's wifi stack today fails utterly, = since there is a single queue per device, rather than one per station).

Whether there are bloat problems at the link level in L= TE due to error correction I don't know yet; but it wouldn't surpri= se me; I know there was in 3g. =A0The people I talked to this morning aren&= #39;t familiar with the HARQ layer in the system.

The base stations are complicated beasts; they have bot= h a linux system in them as well as a real time operating system based devi= ce inside =A0We don't know where the bottle neck(s) are yet. =A0I spent= lunch upping their paranoia and getting them through some conceptual hurdl= es (e.g. multiple bottlenecks that may move, and the like). =A0They will tr= y to get me some of the data so I can help them figure it out. =A0I don'= ;t know if the data flow goes through the linux system in the bnodeb or not= , for example.

Most carriers are now trying to ensure that their backh= auls from the base station are never congested, though that is another know= n source of problems. =A0And then there is the lack of AQM at peering point= routers.... =A0You'd think they might run WRED there, but many/most do= not.
=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0- Jim

=




On Thu, Feb 28, 2013 at 2:08 PM, Dave Taht <dave.taht@gmail.com><= /span> wrote:


On T= hu, Feb 28, 2013 at 1:57 PM, <dpreed@reed.com> wrote:

Doesn't = fq_codel need an estimate of link capacity?=A0


No, it just measures delay. Since so far as I know th= e outgoing portion of LTE is not soft-rate limited, but sensitive to the ac= tual available link bandwidth, fq_codel should work pretty good (if the und= erlying interfaces weren't horribly overbuffired) in that direction.

I'm looking forward to some measurements of actual = buffering at the device driver/device levels.

I do= n't know how inbound to the handset is managed via LTE.

Still quite a few assumptions left to smash in the above.
<= div>
...

in the home router case....=

...

When there are artif= icial rate limits in play (in, for example, a cable modem/CMTS, hooked up v= ia gigE yet rate limiting to 24up/4mbit down), then a rate limiter (tbf,htb= ,hfsc) needs to be applied locally to move that rate limiter/queue manageme= nt into the local device, se we can manage it better.

I'd like to be rid of the need to use htb and come = up with a rate limiter that could be adjusted dynamically from a daemon in = userspace, probing for short all bandwidth fluctuations while monitoring th= e load. It needent send that much data very often, to come up with a stable= result....

You've described one soft-rate sensing scheme (pigg= ybacking on TCP), and I've thought up a few others, that could feed bac= k from a daemon some samples into a a soft(er) rate limiter that would keep= control of the queues in the home router. I am thinking it's going to = take way too long to fix the CPE and far easier to fix the home router via = this method, and certainly it's too painful and inaccurate to merely me= asure the bandwidth once, then set a hard rate, when=A0

So far as I know the gargoyle project was experimenting= with this approach.=A0
=A0
A problem is in places that= connect more than one device to the cable modem... then you end up with th= ose needing to communicate their perception of the actual bandwidth beyond = the link.=A0

Where will it get that from the= 4G or 3G uplink?

=A0

-----Original Message-----
From: "M= aciej Soltysiak" <maciej@soltysiak.com>
Sent: Thursday, February 28, 2013 = 1:03pm
To: cerowrt-devel@lists.bufferbloat.net
Subject: [Cerowrt-devel] Googl= e working on experimental 3.8 Linux kernel for Android

Hiya,
Looks like Google's experimenting with 3.8 for Android: https://android.googlesource.com/kernel/common/+/experim= ental/android-3.8
Sounds great if this means they will utilize fq_codel, TFO, BQL, etc.<= /div>
Anyway my nexus 7 says it has 3.1.10 and this 3.8 will probably go to = Android 5.0 so I hope Nexus 7 will get it too some day=A0or at least 3.3+
Regards,
Maciej

__________________________________= _____________
Cerowrt-devel mailing list
Ce= rowrt-devel@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cerowrt-devel



=

--
Dave T=E4ht

Fixing bufferbloat with cerowrt: <= a href=3D"http://www.teklibre.com/cerowrt/subscribe.html" target=3D"_blank"= >http://www.teklibre.com/cerowrt/subscribe.html=20
_______________________________________________
Cerowrt-devel mailing list
Ce= rowrt-devel@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cerowrt-devel



_______________________________________________
Cerowrt-devel mailing list
Cerowrt-devel@lists.= bufferbloat.net
https://lists.bufferbloat.net/listinfo/cerowrt-devel


--20cf307813fcdf43ac04d6d86439--