From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp131.iad.emailsrvr.com (smtp131.iad.emailsrvr.com [207.97.245.131]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by huchra.bufferbloat.net (Postfix) with ESMTPS id 11A4921F0B3 for ; Thu, 28 Feb 2013 12:58:16 -0800 (PST) Received: from localhost (localhost.localdomain [127.0.0.1]) by smtp23.relay.iad1a.emailsrvr.com (SMTP Server) with ESMTP id C432534A1EA; Thu, 28 Feb 2013 15:58:15 -0500 (EST) X-Virus-Scanned: OK Received: from legacy17.wa-web.iad1a (legacy17.wa-web.iad1a.rsapps.net [192.168.4.107]) by smtp23.relay.iad1a.emailsrvr.com (SMTP Server) with ESMTP id A33CD34A1D0; Thu, 28 Feb 2013 15:58:15 -0500 (EST) Received: from reed.com (localhost.localdomain [127.0.0.1]) by legacy17.wa-web.iad1a (Postfix) with ESMTP id 8D7B63B8032; Thu, 28 Feb 2013 15:58:15 -0500 (EST) Received: by apps.rackspace.com (Authenticated sender: dpreed@reed.com, from: dpreed@reed.com) with HTTP; Thu, 28 Feb 2013 15:58:15 -0500 (EST) Date: Thu, 28 Feb 2013 15:58:15 -0500 (EST) From: dpreed@reed.com To: "Jim Gettys" MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_20130228155815000000_36962" Importance: Normal X-Priority: 3 (Normal) X-Type: html In-Reply-To: References: <1362077845.101114942@apps.rackspace.com> Message-ID: <1362085095.577220022@apps.rackspace.com> X-Mailer: webmail7.0 Cc: "cerowrt-devel@lists.bufferbloat.net" Subject: Re: [Cerowrt-devel] Google working on experimental 3.8 Linux kernel for Android X-BeenThere: cerowrt-devel@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: Development issues regarding the cerowrt test router project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Feb 2013 20:58:17 -0000 ------=_20130228155815000000_36962 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable =0AAt least someone actually saw what I've been seeing for years now in Met= ro area HSPA and LTE deployments.=0A =0AAs you know, when I first reported = this on the e2e list I was told it could not possibly be happening and that= I didn't know what I was talking about. No one in the phone companies was= even interested in replicating my experiments, just dismissing them. It w= as sad.=0A =0AHowever, I had the same experience on the original Honeywell = 6180 dual CPU Multics deployment in about 1973. One day all my benchmarks = were running about 5 times slower every other time I ran the code. I sugge= sted that one of the CPUs was running 5x slower, and it was probably due to= the CPU cache being turned off. The hardware engineer on site said that = that was *impossible*. After 4 more hours of testing, I was sure I was rig= ht. That evening, I got him to take the system down, and we hauled out an = oscilloscope. Sure enough, the gate that received the "cache hit" signal h= ad died in one of the processors. The machine continued to run, since all= that caused was for memory to be fetched every time, rather than using the= cache.=0A =0ABesides the value of finding the "root cause" of anomalies, t= he story points out that you really need to understand software and hardwar= e sometimes. The hardware engineer didn't understand the role of a cache, = even though he fully understood timing margins, TTL logic, core memory (yes= , this machine used core memory), etc.=0A =0AWe both understood oscilloscop= es, fortunately.=0A =0AIn some ways this is like the LTE designers understa= nding TCP. They don't. But sometimes you need to know about both in some= depth.=0A =0ACongratulations, Jim. More Internet Plumbing Merit Badges fo= r you.=0A =0A-----Original Message-----=0AFrom: "Jim Gettys" =0ASent: Thursday, February 28, 2013 3:03pm=0ATo: "Dave Taht" =0ACc: "David P Reed" , "cerowrt-devel@lists= .bufferbloat.net" =0ASubject: Re: [Cer= owrt-devel] Google working on experimental 3.8 Linux kernel for Android=0A= =0A=0A=0AI've got a bit more insight into LTE than I did in the past, court= esy of the last couple days.=0ATo begin with, LTE runs with several classes= of service (the call them bearers). Your VOIP traffic goes into one of th= em.=0AAnd I think there is another as well that is for guaranteed bit rate = traffic. One transmit opportunity may have a bunch of chunks of data, and = that data may be destined for more than one device (IIRC). It's substantia= lly different than WiFi.=0ABut most of what we think of as Internet stuff (= web surfing, dns, etc) all gets dumped into a single best effort ("BE"), cl= ass.=0AThe BE class is definitely badly bloated; I can't say how much becau= se I don't really know yet; the test my colleague ran wasn't run long enoug= h to be confident it filled the buffers). But I will say worse than most c= able modems I've seen. I expect this will be true to different degrees on = different hardware. The other traffic classes haven't been tested yet for = bufferbloat, though I suspect they will have it too. I was told that those= classes have much shorter queues, and when the grow, they dump the whole q= ueues (because delivering late real time traffic is useless). But trust *a= nd* verify.... Verification hasn't been done for anything but BE traffic, = and that hasn't been quantified.=0ABut each device gets a "fair" shot at ba= ndwidth in the cell (or sector of a cell; they run 3 radios in each cell), = where fair is basically time based; if you are at the edge of a cell, you'l= l get a lot less bandwidth than someone near a tower; and this fairness is = guaranteed by a scheduler than runs in the base station (called a b-nodeb, = IIIRC). So the base station guarantees some sort of "fairness" between dev= ices (a place where Linux's wifi stack today fails utterly, since there is = a single queue per device, rather than one per station).=0AWhether there ar= e bloat problems at the link level in LTE due to error correction I don't k= now yet; but it wouldn't surprise me; I know there was in 3g. The people I= talked to this morning aren't familiar with the HARQ layer in the system.= =0AThe base stations are complicated beasts; they have both a linux system = in them as well as a real time operating system based device inside We don= 't know where the bottle neck(s) are yet. I spent lunch upping their paran= oia and getting them through some conceptual hurdles (e.g. multiple bottlen= ecks that may move, and the like). They will try to get me some of the dat= a so I can help them figure it out. I don't know if the data flow goes thr= ough the linux system in the bnodeb or not, for example.=0AMost carriers ar= e now trying to ensure that their backhauls from the base station are never= congested, though that is another known source of problems. And then ther= e is the lack of AQM at peering point routers.... You'd think they might r= un WRED there, but many/most do not.=0A- Jim=0A=0A=0A=0AOn Thu, Feb 28, 201= 3 at 2:08 PM, Dave Taht <[mailto:dave.taht@gmail.com] dave.taht@gmail.com> = wrote:=0A=0A=0A=0A=0AOn Thu, Feb 28, 2013 at 1:57 PM, <[mailto:dpreed@reed= .com] dpreed@reed.com> wrote:=0A=0ADoesn't fq_codel need an estimate of lin= k capacity?=0ANo, it just measures delay. Since so far as I know the outgoi= ng portion of LTE is not soft-rate limited, but sensitive to the actual ava= ilable link bandwidth, fq_codel should work pretty good (if the underlying = interfaces weren't horribly overbuffired) in that direction.=0AI'm looking = forward to some measurements of actual buffering at the device driver/devic= e levels.=0AI don't know how inbound to the handset is managed via LTE.=0A= =0AStill quite a few assumptions left to smash in the above.=0A...=0Ain the= home router case....=0A...=0AWhen there are artificial rate limits in play= (in, for example, a cable modem/CMTS, hooked up via gigE yet rate limiting= to 24up/4mbit down), then a rate limiter (tbf,htb,hfsc) needs to be applie= d locally to move that rate limiter/queue management into the local device,= se we can manage it better.=0AI'd like to be rid of the need to use htb an= d come up with a rate limiter that could be adjusted dynamically from a dae= mon in userspace, probing for short all bandwidth fluctuations while monito= ring the load. It needent send that much data very often, to come up with a= stable result....=0AYou've described one soft-rate sensing scheme (piggyba= cking on TCP), and I've thought up a few others, that could feed back from = a daemon some samples into a a soft(er) rate limiter that would keep contro= l of the queues in the home router. I am thinking it's going to take way to= o long to fix the CPE and far easier to fix the home router via this method= , and certainly it's too painful and inaccurate to merely measure the bandw= idth once, then set a hard rate, when=0ASo far as I know the gargoyle proje= ct was experimenting with this approach.=0A=0AA problem is in places that c= onnect more than one device to the cable modem... then you end up with thos= e needing to communicate their perception of the actual bandwidth beyond th= e link.=0A=0A=0AWhere will it get that from the 4G or 3G uplink?=0A=0A=0A = =0A-----Original Message-----=0AFrom: "Maciej Soltysiak" <[mailto:maciej@so= ltysiak.com] maciej@soltysiak.com>=0ASent: Thursday, February 28, 2013 1:03= pm=0A To: [mailto:cerowrt-devel@lists.bufferbloat.net] cerowrt-devel@lists.= bufferbloat.net=0ASubject: [Cerowrt-devel] Google working on experimental 3= .8 Linux kernel for Android=0A=0A=0A=0AHiya,=0ALooks like Google's experime= nting with 3.8 for Android: [https://android.googlesource.com/kernel/common= /+/experimental/android-3.8] https://android.googlesource.com/kernel/common= /+/experimental/android-3.8=0ASounds great if this means they will utilize = fq_codel, TFO, BQL, etc.=0AAnyway my nexus 7 says it has 3.1.10 and this 3.= 8 will probably go to Android 5.0 so I hope Nexus 7 will get it too some da= y or at least 3.3+=0APhoronix coverage: [http://www.phoronix.com/scan.php?p= age=3Dnews_item&px=3DMTMxMzc] http://www.phoronix.com/scan.php?page=3Dnews_= item&px=3DMTMxMzc=0ATheir 3.8 changelog: [https://android.googlesource.com/= kernel/common/+log/experimental/android-3.8] https://android.googlesource.c= om/kernel/common/+log/experimental/android-3.8=0ARegards,=0AMaciej_________= ______________________________________=0A Cerowrt-devel mailing list=0A[mai= lto:Cerowrt-devel@lists.bufferbloat.net] Cerowrt-devel@lists.bufferbloat.ne= t=0A[https://lists.bufferbloat.net/listinfo/cerowrt-devel] https://lists.bu= fferbloat.net/listinfo/cerowrt-devel=0A=0A=0A=0A-- =0ADave T=C3=A4ht=0A=0AF= ixing bufferbloat with cerowrt: [http://www.teklibre.com/cerowrt/subscribe.= html] http://www.teklibre.com/cerowrt/subscribe.html =0A___________________= ____________________________=0A Cerowrt-devel mailing list=0A[mailto:Cerowr= t-devel@lists.bufferbloat.net] Cerowrt-devel@lists.bufferbloat.net=0A[https= ://lists.bufferbloat.net/listinfo/cerowrt-devel] https://lists.bufferbloat.= net/listinfo/cerowrt-devel=0A=0A ------=_20130228155815000000_36962 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

= At least someone actually saw what I've been seeing for years now in Metro = area HSPA and LTE deployments.

=0A

 = ;

=0A

As you know, when I first reported= this on the e2e list I was told it could not possibly be happening and tha= t I didn't know what I was talking about.  No one in the phone compani= es was even interested in replicating my experiments, just dismissing them.=   It was sad.

=0A

 

=0A

However, I had the same experience on the orig= inal Honeywell 6180 dual CPU Multics deployment in about 1973.  One da= y all my benchmarks were running about 5 times slower every other time I ra= n the code.  I suggested that one of the CPUs was running 5x slower, a= nd it was probably due to the CPU cache being turned off.   The h= ardware engineer on site said that that was *impossible*.  After 4 mor= e hours of testing, I was sure I was right.  That evening, I got him t= o take the system down, and we hauled out an oscilloscope.  Sure enoug= h, the gate that received the "cache hit" signal had died in one of the pro= cessors.   The machine continued to run, since all that caused wa= s for memory to be fetched every time, rather than using the cache.

=0A<= p style=3D"margin:0;padding:0;"> 

=0A

Besides the value of finding the "root cause" of anomalies, the story po= ints out that you really need to understand software and hardware sometimes= .  The hardware engineer didn't understand the role of a cache, even t= hough he fully understood timing margins, TTL logic, core memory (yes, this= machine used core memory), etc.

=0A

&nb= sp;

=0A

We both understood oscilloscopes= , fortunately.

=0A

 

=0A

In some ways this is like the LTE designers unders= tanding TCP.   They don't.  But sometimes you need to know a= bout both in some depth.

=0A

 

= =0A

Congratulations, Jim.  More Intern= et Plumbing Merit Badges for you.

=0A

&n= bsp;

=0A

-----Original Message-----
From: "Jim Gettys" <jg@freedesktop.org>
Sent: Thursday, Februar= y 28, 2013 3:03pm
To: "Dave Taht" <dave.taht@gmail.com>
Cc:= "David P Reed" <dpreed@reed.com>, "cerowrt-devel@lists.bufferbloat.n= et" <cerowrt-devel@lists.bufferbloat.net>
Subject: Re: [Cerowrt-= devel] Google working on experimental 3.8 Linux kernel for Android

=0A
=0A
I've got a b= it more insight into LTE than I did in the past, courtesy of the last coupl= e days.=0A
To begin with, LTE runs with several classes of service (the= call them bearers).  Your VOIP traffic goes into one of them.
= =0A
And I think there is another as well that is for guaranteed bit rat= e traffic.  One transmit opportunity may have a bunch of chunks of dat= a, and that data may be destined for more than one device (IIRC).  It'= s substantially different than WiFi.
=0A
But most of what we think= of as Internet stuff (web surfing, dns, etc) all gets dumped into a single= best effort ("BE"), class.
=0A
The BE class is definitely&nb= sp;badly bloated; I can't say how much because I don't really know yet; the= test my colleague ran wasn't run long enough to be confident it filled the= buffers).  But I will say worse than most cable modems I've seen. &nb= sp;I expect this will be true to different degrees on different hardware. &= nbsp;The other traffic classes haven't been tested yet for bufferbloat, tho= ugh I suspect they will have it too.  I was told that those classes ha= ve much shorter queues, and when the grow, they dump the whole queues (beca= use delivering late real time traffic is useless).  But trust *and* ve= rify....  Verification hasn't been done for anything but BE traffic, a= nd that hasn't been quantified.
=0A
But each device gets a "fair" = shot at bandwidth in the cell (or sector of a cell; they run 3 radios in ea= ch cell), where fair is basically time based; if you are at the edge of a c= ell, you'll get a lot less bandwidth than someone near a tower; and this fa= irness is guaranteed by a scheduler than runs in the base station (called a= b-nodeb, IIIRC).  So the base station guarantees some sort of "fairne= ss" between devices (a place where Linux's wifi stack today fails utterly, = since there is a single queue per device, rather than one per station).=0A
Whether there are bloat problems at the link level in LTE due to = error correction I don't know yet; but it wouldn't surprise me; I know ther= e was in 3g.  The people I talked to this morning aren't familiar with= the HARQ layer in the system.
=0A
The base stations are complicat= ed beasts; they have both a linux system in them as well as a real time ope= rating system based device inside  We don't know where the bottle neck= (s) are yet.  I spent lunch upping their paranoia and getting them thr= ough some conceptual hurdles (e.g. multiple bottlenecks that may move, and = the like).  They will try to get me some of the data so I can help the= m figure it out.  I don't know if the data flow goes through the linux= system in the bnodeb or not, for example.
=0A
Most carriers are n= ow trying to ensure that their backhauls from the base station are never co= ngested, though that is another known source of problems.  And then th= ere is the lack of AQM at peering point routers....  You'd think they = might run WRED there, but many/most do not.
=0A
- Jim
=0A=0A


=0A
O= n Thu, Feb 28, 2013 at 2:08 PM, Dave Taht <dave.taht@gmail.com> wrote:
=0A


=0A=0A
On Thu, Feb 28, 2013 at 1:57 = PM, <dpreed@reed.com> wrote:
=0A
=0A

Doesn't fq_codel need an estimate= of link capacity?

=0A
=0A
=0A
No, it just m= easures delay. Since so far as I know the outgoing portion of LTE is not so= ft-rate limited, but sensitive to the actual available link bandwidth, fq_c= odel should work pretty good (if the underlying interfaces weren't horribly= overbuffired) in that direction.
=0A
I'm looking forward to some = measurements of actual buffering at the device driver/device levels.
= =0A
I don't know how inbound to the handset is managed via LTE.
= =0A
=0A
Still quite a few assumptions left to smash in the ab= ove.
=0A
...
=0A
in the home router case....
=0A...
=0A
When there are artificial rate limits in play (in, for ex= ample, a cable modem/CMTS, hooked up via gigE yet rate limiting to 24up/4mb= it down), then a rate limiter (tbf,htb,hfsc) needs to be applied locally to= move that rate limiter/queue management into the local device, se we can m= anage it better.
=0A
I'd like to be rid of the need to use htb and= come up with a rate limiter that could be adjusted dynamically from a daem= on in userspace, probing for short all bandwidth fluctuations while monitor= ing the load. It needent send that much data very often, to come up with a = stable result....
=0A
You've described one soft-rate sensing schem= e (piggybacking on TCP), and I've thought up a few others, that could feed = back from a daemon some samples into a a soft(er) rate limiter that would k= eep control of the queues in the home router. I am thinking it's going to t= ake way too long to fix the CPE and far easier to fix the home router via t= his method, and certainly it's too painful and inaccurate to merely measure= the bandwidth once, then set a hard rate, when
=0A
So far as I kn= ow the gargoyle project was experimenting with this approach.
=0A
=
=0A
A problem is in places that connect more than one device to t= he cable modem... then you end up with those needing to communicate their p= erception of the actual bandwidth beyond the link.
=0A
=0A
=0A
=0A

Where will it get that from the 4G or 3G uplink?

=0A
=0A
=0A=

 

=0A

-----Original Message-----<= br />From: "Maciej Soltysiak" <maciej@soltysiak.com>
Sent: Thursday, February= 28, 2013 1:03pm
To: cerowrt-devel@lists.bufferbloat.net
Subje= ct: [Cerowrt-devel] Google working on experimental 3.8 Linux kernel for And= roid

=0A
=0A
Hiya,
=0A
Looks like Google's = experimenting with 3.8 for Android: https://an= droid.googlesource.com/kernel/common/+/experimental/android-3.8
= =0A
Sounds great if this means they will utilize fq_codel, TFO, BQL, et= c.
=0A
Anyway my nexus 7 says it has 3.1.10 and this 3.8 will prob= ably go to Android 5.0 so I hope Nexus 7 will get it too some day or a= t least 3.3+
=0A=0ATheir 3.8 changelog: https://android.googl= esource.com/kernel/common/+log/experimental/android-3.8
=0A
Re= gards,
=0A
Maciej
=0A
=0A
=0A
=0A
= =0A
=0A_______________________________________________
Cerowrt-d= evel mailing list
Cerowrt-devel@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/cerowrt-devel

=0A
=0A<= br />
--
Dave T=C3=A4ht

Fixing bufferbloat with cerow= rt: http://www.teklibre.com/cerowrt/subscribe.html
_______________________________________________
Cerowrt-devel maili= ng list
Cerowrt= -devel@lists.bufferbloat.net
https://lists.bufferbloat.net= /listinfo/cerowrt-devel

=0A
=0A
=0A ------=_20130228155815000000_36962--