From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp141.iad.emailsrvr.com (smtp141.iad.emailsrvr.com [207.97.245.141]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by huchra.bufferbloat.net (Postfix) with ESMTPS id 52164200203 for ; Thu, 28 Feb 2013 13:10:07 -0800 (PST) Received: from localhost (localhost.localdomain [127.0.0.1]) by smtp54.relay.iad1a.emailsrvr.com (SMTP Server) with ESMTP id 461782B0F7B; Thu, 28 Feb 2013 16:10:06 -0500 (EST) X-Virus-Scanned: OK Received: from legacy12.wa-web.iad1a (legacy12.wa-web.iad1a.rsapps.net [192.168.4.98]) by smtp54.relay.iad1a.emailsrvr.com (SMTP Server) with ESMTP id 239EA2B0F64; Thu, 28 Feb 2013 16:10:06 -0500 (EST) Received: from reed.com (localhost.localdomain [127.0.0.1]) by legacy12.wa-web.iad1a (Postfix) with ESMTP id 07A0F12856A; Thu, 28 Feb 2013 16:10:06 -0500 (EST) Received: by apps.rackspace.com (Authenticated sender: dpreed@reed.com, from: dpreed@reed.com) with HTTP; Thu, 28 Feb 2013 16:10:06 -0500 (EST) Date: Thu, 28 Feb 2013 16:10:06 -0500 (EST) From: dpreed@reed.com To: "Jim Gettys" MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_20130228161006000000_18827" Importance: Normal X-Priority: 3 (Normal) X-Type: html In-Reply-To: References: <1362077845.101114942@apps.rackspace.com> <1362085095.577220022@apps.rackspace.com> Message-ID: <1362085806.02931925@apps.rackspace.com> X-Mailer: webmail7.0 Cc: "cerowrt-devel@lists.bufferbloat.net" Subject: Re: [Cerowrt-devel] Google working on experimental 3.8 Linux kernel for Android X-BeenThere: cerowrt-devel@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: Development issues regarding the cerowrt test router project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Feb 2013 21:10:07 -0000 ------=_20130228161006000000_18827 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable =0AIt all started when CS departments decided they didn't need EE courses o= r affiliation with EE depts., and continued with the idea that digital comm= unications had nothing to do with the folks who design the gear, so all you= needed to know was the bit layouts of packets in memory to be a "network e= xpert".=0A =0AYou can see this in the curricula at all levels. Cisco certif= ies network people who have never studied control theory, queueing theory, = ..., and the phone companies certify communications engineers who have neve= r run traceroute or ping, much less debugged the performance of a web-based= UI.=0A =0AModularity is great. But it comes at a cost. Besides this kind= of failure, it's the primary cause of security vulnerabilities.=0A =0A----= -Original Message-----=0AFrom: "Jim Gettys" =0ASent: Th= ursday, February 28, 2013 4:02pm=0ATo: "David P Reed" =0AC= c: "Dave Taht" , "cerowrt-devel@lists.bufferbloat.net"= =0ASubject: Re: [Cerowrt-devel] Googl= e working on experimental 3.8 Linux kernel for Android=0A=0A=0A=0AIn short,= people who build hardware devices, or device drivers, don't understand TCP= .=0AThere is a first class education failure in all this.=0A=0AWe have yet = to find almost any device that isn't bloated; the only question is how badl= y.=0A- Jim=0A=0A=0A=0AOn Thu, Feb 28, 2013 at 3:58 PM, <[mailto:dpreed@ree= d.com] dpreed@reed.com> wrote:=0A=0AAt least someone actually saw what I've= been seeing for years now in Metro area HSPA and LTE deployments.=0A =0AAs= you know, when I first reported this on the e2e list I was told it could n= ot possibly be happening and that I didn't know what I was talking about. = No one in the phone companies was even interested in replicating my experim= ents, just dismissing them. It was sad.=0A =0AHowever, I had the same expe= rience on the original Honeywell 6180 dual CPU Multics deployment in about = 1973. One day all my benchmarks were running about 5 times slower every ot= her time I ran the code. I suggested that one of the CPUs was running 5x s= lower, and it was probably due to the CPU cache being turned off. The har= dware engineer on site said that that was *impossible*. After 4 more hours= of testing, I was sure I was right. That evening, I got him to take the s= ystem down, and we hauled out an oscilloscope. Sure enough, the gate that = received the "cache hit" signal had died in one of the processors. The ma= chine continued to run, since all that caused was for memory to be fetched = every time, rather than using the cache.=0A =0ABesides the value of finding= the "root cause" of anomalies, the story points out that you really need t= o understand software and hardware sometimes. The hardware engineer didn't= understand the role of a cache, even though he fully understood timing mar= gins, TTL logic, core memory (yes, this machine used core memory), etc.=0A = =0AWe both understood oscilloscopes, fortunately.=0A =0AIn some ways this i= s like the LTE designers understanding TCP. They don't. But sometimes yo= u need to know about both in some depth.=0A =0ACongratulations, Jim. More = Internet Plumbing Merit Badges for you.=0A=0A=0A =0A-----Original Message--= ---=0AFrom: "Jim Gettys" <[mailto:jg@freedesktop.org] jg@freedesktop.org>= =0ASent: Thursday, February 28, 2013 3:03pm=0ATo: "Dave Taht" <[mailto:dave= .taht@gmail.com] dave.taht@gmail.com>=0A Cc: "David P Reed" <[mailto:dpreed= @reed.com] dpreed@reed.com>, "[mailto:cerowrt-devel@lists.bufferbloat.net] = cerowrt-devel@lists.bufferbloat.net" <[mailto:cerowrt-devel@lists.bufferblo= at.net] cerowrt-devel@lists.bufferbloat.net>=0A Subject: Re: [Cerowrt-devel= ] Google working on experimental 3.8 Linux kernel for Android=0A=0A=0A=0AI'= ve got a bit more insight into LTE than I did in the past, courtesy of the = last couple days.=0ATo begin with, LTE runs with several classes of service= (the call them bearers). Your VOIP traffic goes into one of them.=0AAnd I= think there is another as well that is for guaranteed bit rate traffic. O= ne transmit opportunity may have a bunch of chunks of data, and that data m= ay be destined for more than one device (IIRC). It's substantially differe= nt than WiFi.=0ABut most of what we think of as Internet stuff (web surfing= , dns, etc) all gets dumped into a single best effort ("BE"), class.=0AThe = BE class is definitely badly bloated; I can't say how much because I don't = really know yet; the test my colleague ran wasn't run long enough to be con= fident it filled the buffers). But I will say worse than most cable modems= I've seen. I expect this will be true to different degrees on different h= ardware. The other traffic classes haven't been tested yet for bufferbloat= , though I suspect they will have it too. I was told that those classes ha= ve much shorter queues, and when the grow, they dump the whole queues (beca= use delivering late real time traffic is useless). But trust *and* verify.= ... Verification hasn't been done for anything but BE traffic, and that ha= sn't been quantified.=0ABut each device gets a "fair" shot at bandwidth in = the cell (or sector of a cell; they run 3 radios in each cell), where fair = is basically time based; if you are at the edge of a cell, you'll get a lot= less bandwidth than someone near a tower; and this fairness is guaranteed = by a scheduler than runs in the base station (called a b-nodeb, IIIRC). So= the base station guarantees some sort of "fairness" between devices (a pla= ce where Linux's wifi stack today fails utterly, since there is a single qu= eue per device, rather than one per station).=0AWhether there are bloat pro= blems at the link level in LTE due to error correction I don't know yet; bu= t it wouldn't surprise me; I know there was in 3g. The people I talked to = this morning aren't familiar with the HARQ layer in the system.=0AThe base = stations are complicated beasts; they have both a linux system in them as w= ell as a real time operating system based device inside We don't know wher= e the bottle neck(s) are yet. I spent lunch upping their paranoia and gett= ing them through some conceptual hurdles (e.g. multiple bottlenecks that ma= y move, and the like). They will try to get me some of the data so I can h= elp them figure it out. I don't know if the data flow goes through the lin= ux system in the bnodeb or not, for example.=0AMost carriers are now trying= to ensure that their backhauls from the base station are never congested, = though that is another known source of problems. And then there is the lac= k of AQM at peering point routers.... You'd think they might run WRED ther= e, but many/most do not.=0A- Jim=0A=0A=0A=0AOn Thu, Feb 28, 2013 at 2:08 PM= , Dave Taht <[mailto:dave.taht@gmail.com] dave.taht@gmail.com> wrote:=0A=0A= =0A=0A=0AOn Thu, Feb 28, 2013 at 1:57 PM, <[mailto:dpreed@reed.com] dpreed= @reed.com> wrote:=0A=0ADoesn't fq_codel need an estimate of link capacity?= =0ANo, it just measures delay. Since so far as I know the outgoing portion = of LTE is not soft-rate limited, but sensitive to the actual available link= bandwidth, fq_codel should work pretty good (if the underlying interfaces = weren't horribly overbuffired) in that direction.=0AI'm looking forward to = some measurements of actual buffering at the device driver/device levels.= =0AI don't know how inbound to the handset is managed via LTE.=0AStill quit= e a few assumptions left to smash in the above.=0A...=0Ain the home router = case....=0A...=0AWhen there are artificial rate limits in play (in, for exa= mple, a cable modem/CMTS, hooked up via gigE yet rate limiting to 24up/4mbi= t down), then a rate limiter (tbf,htb,hfsc) needs to be applied locally to = move that rate limiter/queue management into the local device, se we can ma= nage it better.=0AI'd like to be rid of the need to use htb and come up wit= h a rate limiter that could be adjusted dynamically from a daemon in usersp= ace, probing for short all bandwidth fluctuations while monitoring the load= . It needent send that much data very often, to come up with a stable resul= t....=0AYou've described one soft-rate sensing scheme (piggybacking on TCP)= , and I've thought up a few others, that could feed back from a daemon some= samples into a a soft(er) rate limiter that would keep control of the queu= es in the home router. I am thinking it's going to take way too long to fix= the CPE and far easier to fix the home router via this method, and certain= ly it's too painful and inaccurate to merely measure the bandwidth once, th= en set a hard rate, when=0ASo far as I know the gargoyle project was experi= menting with this approach.=0AA problem is in places that connect more than= one device to the cable modem... then you end up with those needing to com= municate their perception of the actual bandwidth beyond the link.=0A=0A=0A= Where will it get that from the 4G or 3G uplink?=0A=0A=0A =0A-----Original = Message-----=0AFrom: "Maciej Soltysiak" <[mailto:maciej@soltysiak.com] maci= ej@soltysiak.com>=0ASent: Thursday, February 28, 2013 1:03pm=0A To: [mailto= :cerowrt-devel@lists.bufferbloat.net] cerowrt-devel@lists.bufferbloat.net= =0ASubject: [Cerowrt-devel] Google working on experimental 3.8 Linux kernel= for Android=0A=0A=0A=0AHiya,=0ALooks like Google's experimenting with 3.8 = for Android: [https://android.googlesource.com/kernel/common/+/experimental= /android-3.8] https://android.googlesource.com/kernel/common/+/experimental= /android-3.8=0ASounds great if this means they will utilize fq_codel, TFO, = BQL, etc.=0AAnyway my nexus 7 says it has 3.1.10 and this 3.8 will probably= go to Android 5.0 so I hope Nexus 7 will get it too some day or at least 3= .3+=0APhoronix coverage: [http://www.phoronix.com/scan.php?page=3Dnews_item= &px=3DMTMxMzc] http://www.phoronix.com/scan.php?page=3Dnews_item&px=3DMTMxM= zc=0ATheir 3.8 changelog: [https://android.googlesource.com/kernel/common/+= log/experimental/android-3.8] https://android.googlesource.com/kernel/commo= n/+log/experimental/android-3.8=0ARegards,=0AMaciej________________________= _______________________=0A Cerowrt-devel mailing list=0A[mailto:Cerowrt-dev= el@lists.bufferbloat.net] Cerowrt-devel@lists.bufferbloat.net=0A[https://li= sts.bufferbloat.net/listinfo/cerowrt-devel] https://lists.bufferbloat.net/l= istinfo/cerowrt-devel=0A=0A=0A=0A-- =0ADave T=C3=A4ht=0A=0AFixing bufferblo= at with cerowrt: [http://www.teklibre.com/cerowrt/subscribe.html] http://ww= w.teklibre.com/cerowrt/subscribe.html =0A _________________________________= ______________=0A Cerowrt-devel mailing list=0A[mailto:Cerowrt-devel@lists.= bufferbloat.net] Cerowrt-devel@lists.bufferbloat.net=0A[https://lists.buffe= rbloat.net/listinfo/cerowrt-devel] https://lists.bufferbloat.net/listinfo/c= erowrt-devel=0A=0A ------=_20130228161006000000_18827 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

= It all started when CS departments decided they didn't need EE courses or a= ffiliation with EE depts., and continued with the idea that digital communi= cations had nothing to do with the folks who design the gear, so all you ne= eded to know was the bit layouts of packets in memory to be a "network expe= rt".

=0A

 

=0A

You can see this in the curricula at all levels. Cisco certi= fies network people who have never studied control theory, queueing theory,= ..., and the phone companies certify communications engineers who have nev= er run traceroute or ping, much less debugged the performance of a web-base= d UI.

=0A

 

=0A

Modularity is great.  But it comes at a cost.  Be= sides this kind of failure, it's the primary cause of security vulnerabilit= ies.

=0A

 

=0A

-----Original Message-----
From: "Jim Gettys" <jg@fr= eedesktop.org>
Sent: Thursday, February 28, 2013 4:02pm
To: "D= avid P Reed" <dpreed@reed.com>
Cc: "Dave Taht" <dave.taht@gma= il.com>, "cerowrt-devel@lists.bufferbloat.net" <cerowrt-devel@lists.b= ufferbloat.net>
Subject: Re: [Cerowrt-devel] Google working on expe= rimental 3.8 Linux kernel for Android

=0A
=0A
In short, people who build hardware devic= es, or device drivers, don't understand TCP.=0A
There is a first class = education failure in all this.
=0A
We have yet to find almost any = device that isn't bloated; the only question is how badly.
=0A
- J= im
=0A
=0A
=0A


=0AOn Thu, Feb 28, 2013 at 3:58 PM, = <dpreed@reed.com> wrote:
=0A
=0A

At least someone actually saw what I've been seeing for = years now in Metro area HSPA and LTE deployments.

=0A

 

=0A

As you know, when I first reported this on the= e2e list I was told it could not possibly be happening and that I didn't k= now what I was talking about.  No one in the phone companies was even = interested in replicating my experiments, just dismissing them.  It wa= s sad.

=0A

 <= /p>=0A

However, I had= the same experience on the original Honeywell 6180 dual CPU Multics deploy= ment in about 1973.  One day all my benchmarks were running about 5 ti= mes slower every other time I ran the code.  I suggested that one of t= he CPUs was running 5x slower, and it was probably due to the CPU cache bei= ng turned off.   The hardware engineer on site said that that was= *impossible*.  After 4 more hours of testing, I was sure I was right.=   That evening, I got him to take the system down, and we hauled out a= n oscilloscope.  Sure enough, the gate that received the "cache hit" s= ignal had died in one of the processors.   The machine continued = to run, since all that caused was for memory to be fetched every time, rath= er than using the cache.

=0A

 

=0A

Besides the value of finding the "root cause" of anomalies, the story p= oints out that you really need to understand software and hardware sometime= s.  The hardware engineer didn't understand the role of a cache, even = though he fully understood timing margins, TTL logic, core memory (yes, thi= s machine used core memory), etc.

=0A

 

=0A

We both understood oscilloscopes, fortunately.

=0A

 

=0A

In some ways this is like the LTE d= esigners understanding TCP.   They don't.  But sometimes you= need to know about both in some depth.

=0A

 

=0A

Congratulations, Jim.  More Internet Plumbing Merit= Badges for you.

=0A
=0A
=0A

 

=0A

-----Original Message-----
From: "Jim Gettys"= <jg@freedesktop= .org>
Sent: Thursday, February 28, 2013 3:03pm
To: "Dave T= aht" <dave.taht= @gmail.com>
Cc: "David P Reed" <dpreed@reed.com>, "cerowrt-devel@lists.buffe= rbloat.net" <cerowrt-devel@lists.bufferbloat.net>
Subject= : Re: [Cerowrt-devel] Google working on experimental 3.8 Linux kernel for A= ndroid

=0A
=0A
I've got a bit more insig= ht into LTE than I did in the past, courtesy of the last couple days.=0ATo begin with, LTE runs with several classes of service (the call them be= arers).  Your VOIP traffic goes into one of them.
=0A
And I t= hink there is another as well that is for guaranteed bit rate traffic. &nbs= p;One transmit opportunity may have a bunch of chunks of data, and that dat= a may be destined for more than one device (IIRC).  It's substantially= different than WiFi.
=0A
But most of what we think of as Internet= stuff (web surfing, dns, etc) all gets dumped into a single best effort ("= BE"), class.
=0A
The BE class is definitely badly bloate= d; I can't say how much because I don't really know yet; the test my collea= gue ran wasn't run long enough to be confident it filled the buffers). &nbs= p;But I will say worse than most cable modems I've seen.  I expect thi= s will be true to different degrees on different hardware.  The other = traffic classes haven't been tested yet for bufferbloat, though I suspect t= hey will have it too.  I was told that those classes have much shorter= queues, and when the grow, they dump the whole queues (because delivering = late real time traffic is useless).  But trust *and* verify....  = Verification hasn't been done for anything but BE traffic, and that hasn't = been quantified.
=0A
But each device gets a "fair" shot at bandwid= th in the cell (or sector of a cell; they run 3 radios in each cell), where= fair is basically time based; if you are at the edge of a cell, you'll get= a lot less bandwidth than someone near a tower; and this fairness is guara= nteed by a scheduler than runs in the base station (called a b-nodeb, IIIRC= ).  So the base station guarantees some sort of "fairness" between dev= ices (a place where Linux's wifi stack today fails utterly, since there is = a single queue per device, rather than one per station).
=0A
Wheth= er there are bloat problems at the link level in LTE due to error correctio= n I don't know yet; but it wouldn't surprise me; I know there was in 3g. &n= bsp;The people I talked to this morning aren't familiar with the HARQ layer= in the system.
=0A
The base stations are complicated beasts; they= have both a linux system in them as well as a real time operating system b= ased device inside  We don't know where the bottle neck(s) are yet. &n= bsp;I spent lunch upping their paranoia and getting them through some conce= ptual hurdles (e.g. multiple bottlenecks that may move, and the like). &nbs= p;They will try to get me some of the data so I can help them figure it out= .  I don't know if the data flow goes through the linux system in the = bnodeb or not, for example.
=0A
Most carriers are now trying to en= sure that their backhauls from the base station are never congested, though= that is another known source of problems.  And then there is the lack= of AQM at peering point routers....  You'd think they might run WRED = there, but many/most do not.
=0A
- Jim
=0A
=0A


=0A
On Thu, Feb 28, = 2013 at 2:08 PM, Dave Taht <dave.taht@gmail.com> wrote:=0A


=0A
=0A
On Thu, Feb 28, 2013 at 1:57 PM, <<= a href=3D"mailto:dpreed@reed.com" target=3D"_blank">dpreed@reed.com>= wrote:
=0A
=0A

Doesn't fq_codel need an estimate of link capacity?

=0A
=0A
=0A
No, it just measures delay. Since so far = as I know the outgoing portion of LTE is not soft-rate limited, but sensiti= ve to the actual available link bandwidth, fq_codel should work pretty good= (if the underlying interfaces weren't horribly overbuffired) in that direc= tion.
=0A
I'm looking forward to some measurements of actual buffe= ring at the device driver/device levels.
=0A
I don't know how inbo= und to the handset is managed via LTE.
=0A
Still quite a few assum= ptions left to smash in the above.
=0A
...
=0A
in the hom= e router case....
=0A
...
=0A
When there are artificial r= ate limits in play (in, for example, a cable modem/CMTS, hooked up via gigE= yet rate limiting to 24up/4mbit down), then a rate limiter (tbf,htb,hfsc) = needs to be applied locally to move that rate limiter/queue management into= the local device, se we can manage it better.
=0A
I'd like to be = rid of the need to use htb and come up with a rate limiter that could be ad= justed dynamically from a daemon in userspace, probing for short all bandwi= dth fluctuations while monitoring the load. It needent send that much data = very often, to come up with a stable result....
=0A
You've describ= ed one soft-rate sensing scheme (piggybacking on TCP), and I've thought up = a few others, that could feed back from a daemon some samples into a a soft= (er) rate limiter that would keep control of the queues in the home router.= I am thinking it's going to take way too long to fix the CPE and far easie= r to fix the home router via this method, and certainly it's too painful an= d inaccurate to merely measure the bandwidth once, then set a hard rate, wh= en
=0A
So far as I know the gargoyle project was experimenting wit= h this approach.
=0A
A problem is in places that connect more than= one device to the cable modem... then you end up with those needing to com= municate their perception of the actual bandwidth beyond the link.
=0A=
=0A
=0A
=0A

Where will it get that from the 4G or 3G uplink?

=0A
=0A
= =0A

 

=0A

-----Original Message---= --
From: "Maciej Soltysiak" <maciej@soltysiak.com>
Sent: Thursday, Febru= ary 28, 2013 1:03pm
To: cerowrt-devel@lists.bufferbloat.net
Su= bject: [Cerowrt-devel] Google working on experimental 3.8 Linux kernel for = Android

=0A
=0A
Hiya,
=0A
Looks like Google= 's experimenting with 3.8 for Android: https:/= /android.googlesource.com/kernel/common/+/experimental/android-3.8=0A
Sounds great if this means they will utilize fq_codel, TFO, BQL, e= tc.
=0A
Anyway my nexus 7 says it has 3.1.10 and this 3.8 will pro= bably go to Android 5.0 so I hope Nexus 7 will get it too some day or = at least 3.3+
=0A=0ATheir 3.8 changelog: https://android.goog= lesource.com/kernel/common/+log/experimental/android-3.8
=0A
R= egards,
=0A
Maciej
=0A
=0A
=0A
=0A=0A
=0A_______________________________________________
Cerowrt-= devel mailing list
Cerowrt-devel@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cerowrt-devel

=0A
=0A

-- Dave T=C3=A4ht

Fixing bufferbloat with cerowrt: http://www.= teklibre.com/cerowrt/subscribe.html
______________= _________________________________
Cerowrt-devel mailing list
Cerow= rt-devel@lists.bufferbloat.net
https://lists.bufferbloat.n= et/listinfo/cerowrt-devel

=0A
=0A
=0A=
=0A
=0A
=0A
=0A
=0A
=0A<= /font> ------=_20130228161006000000_18827--