From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-oa0-f48.google.com (mail-oa0-f48.google.com [209.85.219.48]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority" (verified OK)) by huchra.bufferbloat.net (Postfix) with ESMTPS id 2734C21F0E8 for ; Thu, 28 Feb 2013 13:02:31 -0800 (PST) Received: by mail-oa0-f48.google.com with SMTP id j1so4403868oag.7 for ; Thu, 28 Feb 2013 13:02:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=z0KfGsY1k2VoJn0983OxZCBe7hzMKZN3SrqkSbR213Q=; b=E3CJv6Svcw96tYOA13I/CIzqVCZw227Zz8fA3IJjcuP8SG9d7bnWQ3vydO2mB58zCr BOTzTVxkC4+/PkHpDZE2bStXrUED2lb83m0Qiew5ujPcXs9Y91j1eAQNnIRf34QYPtm2 2fvl/vgv2Wwwe9tup6rfW8rg3snuuM/VxCOJ0+AGRJnNO7QM8OWzjeSsP6IQr6wOx4+k M7C+rAMtIgamCIexZjco+m30mTNtOtnWuLDAynXtkZy4EjfFTii5fgrhR8WIhx7A/1ry NaiIioiDZ6GUt4ock8KtCPezOIsisDmzQzEJBtJdDUEqLNOlVoAOp0j3dY49E1dzzjKe CSkA== MIME-Version: 1.0 X-Received: by 10.60.3.71 with SMTP id a7mr6754657oea.35.1362085350219; Thu, 28 Feb 2013 13:02:30 -0800 (PST) Sender: gettysjim@gmail.com Received: by 10.76.22.193 with HTTP; Thu, 28 Feb 2013 13:02:30 -0800 (PST) In-Reply-To: <1362085095.577220022@apps.rackspace.com> References: <1362077845.101114942@apps.rackspace.com> <1362085095.577220022@apps.rackspace.com> Date: Thu, 28 Feb 2013 16:02:30 -0500 X-Google-Sender-Auth: qHyNP4hoT9AQr0y1SaT1Dn0p0dM Message-ID: From: Jim Gettys To: David P Reed Content-Type: multipart/alternative; boundary=e89a8f839d5174284c04d6cf351c Cc: "cerowrt-devel@lists.bufferbloat.net" Subject: Re: [Cerowrt-devel] Google working on experimental 3.8 Linux kernel for Android X-BeenThere: cerowrt-devel@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: Development issues regarding the cerowrt test router project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Feb 2013 21:02:31 -0000 --e89a8f839d5174284c04d6cf351c Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable In short, people who build hardware devices, or device drivers, don't understand TCP. There is a first class education failure in all this. We have yet to find almost any device that isn't bloated; the only question is how badly. - Jim On Thu, Feb 28, 2013 at 3:58 PM, wrote: > At least someone actually saw what I've been seeing for years now in Metr= o > area HSPA and LTE deployments. > > > > As you know, when I first reported this on the e2e list I was told it > could not possibly be happening and that I didn't know what I was talking > about. No one in the phone companies was even interested in replicating = my > experiments, just dismissing them. It was sad. > > > > However, I had the same experience on the original Honeywell 6180 dual CP= U > Multics deployment in about 1973. One day all my benchmarks were running > about 5 times slower every other time I ran the code. I suggested that o= ne > of the CPUs was running 5x slower, and it was probably due to the CPU cac= he > being turned off. The hardware engineer on site said that that was > *impossible*. After 4 more hours of testing, I was sure I was right. Th= at > evening, I got him to take the system down, and we hauled out an > oscilloscope. Sure enough, the gate that received the "cache hit" signal > had died in one of the processors. The machine continued to run, since > all that caused was for memory to be fetched every time, rather than usin= g > the cache. > > > > Besides the value of finding the "root cause" of anomalies, the story > points out that you really need to understand software and hardware > sometimes. The hardware engineer didn't understand the role of a cache, > even though he fully understood timing margins, TTL logic, core memory > (yes, this machine used core memory), etc. > > > > We both understood oscilloscopes, fortunately. > > > > In some ways this is like the LTE designers understanding TCP. They > don't. But sometimes you need to know about both in some depth. > > > > Congratulations, Jim. More Internet Plumbing Merit Badges for you. > > > > -----Original Message----- > From: "Jim Gettys" > Sent: Thursday, February 28, 2013 3:03pm > To: "Dave Taht" > Cc: "David P Reed" , "cerowrt-devel@lists.bufferbloat.ne= t" > > Subject: Re: [Cerowrt-devel] Google working on experimental 3.8 Linux > kernel for Android > > I've got a bit more insight into LTE than I did in the past, courtesy of > the last couple days. > To begin with, LTE runs with several classes of service (the call them > bearers). Your VOIP traffic goes into one of them. > And I think there is another as well that is for guaranteed bit rate > traffic. One transmit opportunity may have a bunch of chunks of data, an= d > that data may be destined for more than one device (IIRC). It's > substantially different than WiFi. > But most of what we think of as Internet stuff (web surfing, dns, etc) al= l > gets dumped into a single best effort ("BE"), class. > The BE class is definitely badly bloated; I can't say how much because I > don't really know yet; the test my colleague ran wasn't run long enough t= o > be confident it filled the buffers). But I will say worse than most cabl= e > modems I've seen. I expect this will be true to different degrees on > different hardware. The other traffic classes haven't been tested yet fo= r > bufferbloat, though I suspect they will have it too. I was told that tho= se > classes have much shorter queues, and when the grow, they dump the whole > queues (because delivering late real time traffic is useless). But trust > *and* verify.... Verification hasn't been done for anything but BE > traffic, and that hasn't been quantified. > But each device gets a "fair" shot at bandwidth in the cell (or sector of > a cell; they run 3 radios in each cell), where fair is basically time > based; if you are at the edge of a cell, you'll get a lot less bandwidth > than someone near a tower; and this fairness is guaranteed by a scheduler > than runs in the base station (called a b-nodeb, IIIRC). So the base > station guarantees some sort of "fairness" between devices (a place where > Linux's wifi stack today fails utterly, since there is a single queue per > device, rather than one per station). > Whether there are bloat problems at the link level in LTE due to error > correction I don't know yet; but it wouldn't surprise me; I know there wa= s > in 3g. The people I talked to this morning aren't familiar with the HARQ > layer in the system. > The base stations are complicated beasts; they have both a linux system i= n > them as well as a real time operating system based device inside We don'= t > know where the bottle neck(s) are yet. I spent lunch upping their parano= ia > and getting them through some conceptual hurdles (e.g. multiple bottlenec= ks > that may move, and the like). They will try to get me some of the data s= o > I can help them figure it out. I don't know if the data flow goes throug= h > the linux system in the bnodeb or not, for example. > Most carriers are now trying to ensure that their backhauls from the base > station are never congested, though that is another known source of > problems. And then there is the lack of AQM at peering point routers.... > You'd think they might run WRED there, but many/most do not. > - Jim > > > On Thu, Feb 28, 2013 at 2:08 PM, Dave Taht wrote: > >> >> >> On Thu, Feb 28, 2013 at 1:57 PM, wrote: >> >>> Doesn't fq_codel need an estimate of link capacity? >>> >> No, it just measures delay. Since so far as I know the outgoing portion >> of LTE is not soft-rate limited, but sensitive to the actual available l= ink >> bandwidth, fq_codel should work pretty good (if the underlying interface= s >> weren't horribly overbuffired) in that direction. >> I'm looking forward to some measurements of actual buffering at the >> device driver/device levels. >> I don't know how inbound to the handset is managed via LTE. >> Still quite a few assumptions left to smash in the above. >> ... >> in the home router case.... >> ... >> When there are artificial rate limits in play (in, for example, a cable >> modem/CMTS, hooked up via gigE yet rate limiting to 24up/4mbit down), th= en >> a rate limiter (tbf,htb,hfsc) needs to be applied locally to move that r= ate >> limiter/queue management into the local device, se we can manage it bett= er. >> I'd like to be rid of the need to use htb and come up with a rate limite= r >> that could be adjusted dynamically from a daemon in userspace, probing f= or >> short all bandwidth fluctuations while monitoring the load. It needent s= end >> that much data very often, to come up with a stable result.... >> You've described one soft-rate sensing scheme (piggybacking on TCP), and >> I've thought up a few others, that could feed back from a daemon some >> samples into a a soft(er) rate limiter that would keep control of the >> queues in the home router. I am thinking it's going to take way too long= to >> fix the CPE and far easier to fix the home router via this method, and >> certainly it's too painful and inaccurate to merely measure the bandwidt= h >> once, then set a hard rate, when >> So far as I know the gargoyle project was experimenting with this >> approach. >> A problem is in places that connect more than one device to the cable >> modem... then you end up with those needing to communicate their percept= ion >> of the actual bandwidth beyond the link. >> >>> Where will it get that from the 4G or 3G uplink? >>> >>> >>> >>> -----Original Message----- >>> From: "Maciej Soltysiak" >>> Sent: Thursday, February 28, 2013 1:03pm >>> To: cerowrt-devel@lists.bufferbloat.net >>> Subject: [Cerowrt-devel] Google working on experimental 3.8 Linux kerne= l >>> for Android >>> >>> Hiya, >>> Looks like Google's experimenting with 3.8 for Android: >>> https://android.googlesource.com/kernel/common/+/experimental/android-3= .8 >>> Sounds great if this means they will utilize fq_codel, TFO, BQL, etc. >>> Anyway my nexus 7 says it has 3.1.10 and this 3.8 will probably go to >>> Android 5.0 so I hope Nexus 7 will get it too some day or at least 3.3+ >>> Phoronix coverage: >>> http://www.phoronix.com/scan.php?page=3Dnews_item&px=3DMTMxMzc >>> Their 3.8 changelog: >>> https://android.googlesource.com/kernel/common/+log/experimental/androi= d-3.8 >>> Regards, >>> Maciej >>> _______________________________________________ >>> Cerowrt-devel mailing list >>> Cerowrt-devel@lists.bufferbloat.net >>> https://lists.bufferbloat.net/listinfo/cerowrt-devel >>> >>> >> >> -- >> Dave T=E4ht >> >> Fixing bufferbloat with cerowrt: >> http://www.teklibre.com/cerowrt/subscribe.html >> _______________________________________________ >> Cerowrt-devel mailing list >> Cerowrt-devel@lists.bufferbloat.net >> https://lists.bufferbloat.net/listinfo/cerowrt-devel >> >> --e89a8f839d5174284c04d6cf351c Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
In short, people who build hardware devices, or device dri= vers, don't understand TCP.

There is a first class e= ducation failure in all this.

We have yet to f= ind almost any device that isn't bloated; the only question is how badl= y.
=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0- Jim



On Thu, Feb 28, 2013 at= 3:58 PM, <dpreed@reed.com> wrote:

At least someone actually saw what I've been seeing = for years now in Metro area HSPA and LTE deployments.

=A0

As you know, when I first reported this on = the e2e list I was told it could not possibly be happening and that I didn&= #39;t know what I was talking about.=A0 No one in the phone companies was e= ven interested in replicating my experiments, just dismissing them.=A0 It w= as sad.

=A0

However, I had the same experience on the o= riginal Honeywell 6180 dual CPU Multics deployment in about 1973.=A0 One da= y all my benchmarks were running about 5 times slower every other time I ra= n the code.=A0 I suggested that one of the CPUs was running 5x slower, and = it was probably due to the CPU cache being turned off.=A0=A0 The hardware e= ngineer on site said that that was *impossible*.=A0 After 4 more hours of t= esting, I was sure I was right.=A0 That evening, I got him to take the syst= em down, and we hauled out an oscilloscope.=A0 Sure enough, the gate that r= eceived the "cache hit" signal had died in one of the processors.= =A0=A0 The machine continued to run, since all that caused was for memory t= o be fetched every time, rather than using the cache.

=A0

Besides the value of finding the "root= cause" of anomalies, the story points out that you really need to und= erstand software and hardware sometimes.=A0 The hardware engineer didn'= t understand the role of a cache, even though he fully understood timing ma= rgins, TTL logic, core memory (yes, this machine used core memory), etc.

=A0

We both understood oscilloscopes, fortunate= ly.

=A0

In some ways this is like the LTE designers= understanding TCP.=A0=A0 They don't.=A0 But sometimes you need to know= about both in some depth.

=A0

Congratulations, Jim.=A0 More Internet Plum= bing Merit Badges for you.

=A0

-----Original Message-----
From: "J= im Gettys" <jg@freedesktop.org>
Sent: Thursday, February 28, 2013 3:03pm
= To: "Dave Taht" <dave.taht@gmail.com>
Cc: "David P Reed" <dpreed@reed.com>, "cerowrt-devel@lists.bufferbloat.ne= t" <cerowrt-devel@lists.bufferbloat.net>
Subject: Re: [Cerowrt-devel] Google working on experimental 3.8 Linux kerne= l for Android

I've got a bit more insight into LTE than I did in the= past, courtesy of the last couple days.
To begin with, LTE runs with several classes of service (the call them= bearers). =A0Your VOIP traffic goes into one of them.
And I think there is another as well that is for guaranteed bit rate t= raffic. =A0One transmit opportunity may have a bunch of chunks of data, and= that data may be destined for more than one device (IIRC). =A0It's sub= stantially different than WiFi.
But most of what we think of as Internet stuff (web surfing, dns, etc)= all gets dumped into a single best effort ("BE"), class.
The BE class is=A0definitely=A0badly bloated; I can't say how much= because I don't really know yet; the test my colleague ran wasn't = run long enough to be confident it filled the buffers). =A0But I will say w= orse than most cable modems I've seen. =A0I expect this will be true to= different degrees on different hardware. =A0The other traffic classes have= n't been tested yet for bufferbloat, though I suspect they will have it= too. =A0I was told that those classes have much shorter queues, and when t= he grow, they dump the whole queues (because delivering late real time traf= fic is useless). =A0But trust *and* verify.... =A0Verification hasn't b= een done for anything but BE traffic, and that hasn't been quantified.<= /div>
But each device gets a "fair" shot at bandwidth in the cell = (or sector of a cell; they run 3 radios in each cell), where fair is basica= lly time based; if you are at the edge of a cell, you'll get a lot less= bandwidth than someone near a tower; and this fairness is guaranteed by a = scheduler than runs in the base station (called a b-nodeb, IIIRC). =A0So th= e base station guarantees some sort of "fairness" between devices= (a place where Linux's wifi stack today fails utterly, since there is = a single queue per device, rather than one per station).
Whether there are bloat problems at the link level in LTE due to error= correction I don't know yet; but it wouldn't surprise me; I know t= here was in 3g. =A0The people I talked to this morning aren't familiar = with the HARQ layer in the system.
The base stations are complicated beasts; they have both a linux syste= m in them as well as a real time operating system based device inside =A0We= don't know where the bottle neck(s) are yet. =A0I spent lunch upping t= heir paranoia and getting them through some conceptual hurdles (e.g. multip= le bottlenecks that may move, and the like). =A0They will try to get me som= e of the data so I can help them figure it out. =A0I don't know if the = data flow goes through the linux system in the bnodeb or not, for example.<= /div>
Most carriers are now trying to ensure that their backhauls from the b= ase station are never congested, though that is another known source of pro= blems. =A0And then there is the lack of AQM at peering point routers.... = =A0You'd think they might run WRED there, but many/most do not.
- Jim


On Thu, Feb 28, 2013 at 2:08 PM, Dave Taht <d= ave.taht@gmail.com> wrote:


On Thu, Feb 28, 2013 at 1:57 PM, <dpreed@reed.com> wrote= :

Doesn't fq_codel nee= d an estimate of link capacity?

No, it just measures delay. Since so far as I know the outgoing portio= n of LTE is not soft-rate limited, but sensitive to the actual available li= nk bandwidth, fq_codel should work pretty good (if the underlying interface= s weren't horribly overbuffired) in that direction.
I'm looking forward to some measurements of actual buffering at th= e device driver/device levels.
I don't know how inbound to the handset is managed via LTE.
Still quite a few assumptions left to smash in the above.
...
in the home router case....
...
When there are artificial rate limits in play (in, for example, a cabl= e modem/CMTS, hooked up via gigE yet rate limiting to 24up/4mbit down), the= n a rate limiter (tbf,htb,hfsc) needs to be applied locally to move that ra= te limiter/queue management into the local device, se we can manage it bett= er.
I'd like to be rid of the need to use htb and come up with a rate = limiter that could be adjusted dynamically from a daemon in userspace, prob= ing for short all bandwidth fluctuations while monitoring the load. It need= ent send that much data very often, to come up with a stable result....
You've described one soft-rate sensing scheme (piggybacking on TCP= ), and I've thought up a few others, that could feed back from a daemon= some samples into a a soft(er) rate limiter that would keep control of the= queues in the home router. I am thinking it's going to take way too lo= ng to fix the CPE and far easier to fix the home router via this method, an= d certainly it's too painful and inaccurate to merely measure the bandw= idth once, then set a hard rate, when
So far as I know the gargoyle project was experimenting with this appr= oach.
A problem is in places that connect more than one device to the cable = modem... then you end up with those needing to communicate their perception= of the actual bandwidth beyond the link.

Where will it get that f= rom the 4G or 3G uplink?

=A0

-----Original Message---= --
From: "Maciej Soltysiak" <maciej@soltysiak.com>
Sent: Thursday,= February 28, 2013 1:03pm
To: cerowrt-devel@lists.bufferbloat.net
Subject: [Cerowrt-devel] Goog= le working on experimental 3.8 Linux kernel for Android

Hiya,
Looks like Google's experimenting with 3.8 for Android: https://android.googlesource.com/kernel/common/+/experim= ental/android-3.8
Sounds great if this means they will utilize fq_codel, TFO, BQL, etc.<= /div>
Anyway my nexus 7 says it has 3.1.10 and this 3.8 will probably go to = Android 5.0 so I hope Nexus 7 will get it too some day=A0or at least 3.3+
Regards,
Maciej
_______________________________________________
Cerowrt-devel mailing l= ist
Cerowrt-devel@lists.bufferbloat.net
https://lists.buffe= rbloat.net/listinfo/cerowrt-devel



--
Dave T=E4ht

Fixin= g bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.htm= l
_______________________________________________
Cerowrt-devel mailing l= ist
Cerowrt-devel@lists.bufferbloat.net
https://lists.buffe= rbloat.net/listinfo/cerowrt-devel


--e89a8f839d5174284c04d6cf351c--