From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <dpreed@reed.com>
Received: from smtp201.iad.emailsrvr.com (smtp201.iad.emailsrvr.com
	[207.97.245.201])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(Client did not present a certificate)
	by huchra.bufferbloat.net (Postfix) with ESMTPS id 59A7721F0E6
	for <cerowrt-devel@lists.bufferbloat.net>;
	Mon, 20 Aug 2012 19:33:19 -0700 (PDT)
Received: from localhost (localhost.localdomain [127.0.0.1])
	by smtp40.relay.iad1a.emailsrvr.com (SMTP Server) with ESMTP id
	EDEA73501FD; Mon, 20 Aug 2012 22:33:17 -0400 (EDT)
X-Virus-Scanned: OK
Received: from legacy13.wa-web.iad1a (legacy13.wa-web.iad1a.rsapps.net
	[192.168.4.99])
	by smtp40.relay.iad1a.emailsrvr.com (SMTP Server) with ESMTP id
	C59E33500BB; Mon, 20 Aug 2012 22:33:17 -0400 (EDT)
Received: from reed.com (localhost.localdomain [127.0.0.1])
	by legacy13.wa-web.iad1a (Postfix) with ESMTP id A27E7370409;
	Mon, 20 Aug 2012 22:33:17 -0400 (EDT)
Received: by apps.rackspace.com
	(Authenticated sender: dpreed@reed.com, from: dpreed@reed.com) 
	with HTTP; Mon, 20 Aug 2012 22:33:17 -0400 (EDT)
Date: Mon, 20 Aug 2012 22:33:17 -0400 (EDT)
From: dpreed@reed.com
To: "Sebastian Moeller" <moeller0@gmx.de>
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_20120820223317000000_65240"
Importance: Normal
X-Priority: 3 (Normal)
X-Type: html
In-Reply-To: <2998C331-777B-41B3-A6BC-8285461EF729@gmx.de>
References: <CAA93jw50MeqWH6TVKditFGfg-V-mOi-UUtsABqd+WHs2vedHQw@mail.gmail.com>
	<36D61FDC-9AA9-46CC-ACBB-2D28B250C660@gmx.de> 
	<CAA93jw5JV0L0MLYxuO-ok2joOkodie_JkxNs_VAJat_eYhV1gw@mail.gmail.com> 
	<2998C331-777B-41B3-A6BC-8285461EF729@gmx.de>
Message-ID: <1345516397.664231592@apps.rackspace.com>
X-Mailer: webmail7.0
Cc: cerowrt-devel@lists.bufferbloat.net
Subject: Re: [Cerowrt-devel] cerowrt 3.3.8-17 is released
X-BeenThere: cerowrt-devel@lists.bufferbloat.net
X-Mailman-Version: 2.1.13
Precedence: list
List-Id: Development issues regarding the cerowrt test router project
	<cerowrt-devel.lists.bufferbloat.net>
List-Unsubscribe: <https://lists.bufferbloat.net/options/cerowrt-devel>,
	<mailto:cerowrt-devel-request@lists.bufferbloat.net?subject=unsubscribe>
List-Archive: <https://lists.bufferbloat.net/pipermail/cerowrt-devel>
List-Post: <mailto:cerowrt-devel@lists.bufferbloat.net>
List-Help: <mailto:cerowrt-devel-request@lists.bufferbloat.net?subject=help>
List-Subscribe: <https://lists.bufferbloat.net/listinfo/cerowrt-devel>,
	<mailto:cerowrt-devel-request@lists.bufferbloat.net?subject=subscribe>
X-List-Received-Date: Tue, 21 Aug 2012 02:33:19 -0000

------=_20120820223317000000_65240
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

=0AI'm confused.  Fq_codel does not (to my knowledge) fix bufferbloat *in y=
our cable modem*  or *in the CMTS head-end*.=0A =0AHow could it?   In order=
 for that to be fixed, you need to manage the buffer in the cable modem its=
elf.=0A =0A-----Original Message-----=0AFrom: "Sebastian Moeller" <moeller0=
@gmx.de>=0ASent: Monday, August 20, 2012 2:24pm=0ATo: "Dave Taht" <dave.tah=
t@gmail.com>=0ACc: cerowrt-devel@lists.bufferbloat.net=0ASubject: Re: [Cero=
wrt-devel] cerowrt 3.3.8-17 is released=0A=0A=0A=0AHi Dave,=0A=0Aso I went =
to play around with this a bit more. I turned to UDP flooding my cable mode=
m through the router and this surely allows me to create enough load on the=
 wndr3700v2 to cause the allocation errors and as a "bonus" also to drive t=
he router to reboot (driven by the watchdog timer?). Here is the script I u=
sed over 5G wireless (from http://blog.ioshints.info/2008/03/udp-flood-in-p=
erl.html)=0A=0A#!/usr/bin/perl=0A##############=0A=0A# udp flood.=0A#######=
#######=0A=0Ause Socket;=0Ause strict;=0A=0Aif ($#ARGV !=3D 3) {=0A print "=
flood.pl <ip> <port> <size> <time>\n\n";=0A print " port=3D0: use random po=
rts\n";=0A print " size=3D0: use random size between 64 and 1024\n";=0A pri=
nt " time=3D0: continuous flood\n";=0A exit(1);=0A}=0A=0Amy ($ip,$port,$siz=
e,$time) =3D @ARGV;=0A=0Amy ($iaddr,$endtime,$psize,$pport);=0A=0A$iaddr =
=3D inet_aton("$ip") or die "Cannot resolve hostname $ip\n";=0A$endtime =3D=
 time() + ($time ? $time : 1000000);=0A=0Asocket(flood, PF_INET, SOCK_DGRAM=
, 17);=0A=0A=0Aprint "Flooding $ip " . ($port ? $port : "random") . " port =
with " . =0A ($size ? "$size-byte" : "random size") . " packets" . =0A ($ti=
me ? " for $time seconds" : "") . "\n";=0Aprint "Break with Ctrl-C\n" unles=
s $time;=0A=0Afor (;time() <=3D $endtime;) {=0A $psize =3D $size ? $size : =
int(rand(1024-64)+64) ;=0A $pport =3D $port ? $port : int(rand(65500))+1;=
=0A=0A send(flood, pack("a$psize","flood"), 0, pack_sockaddr_in($pport, $ia=
ddr));}=0A=0Acalled as either=0Audp_flood.pl 192.168.100.1 0 1024 240=0Aor =
=0Audp_flood.pl 192.168.100.1 32000 1024 240=0A=0AThe first version with ra=
ndomized port number spreads the load nicely over many fq_codel bins/flows =
and seems slightly more likely to cause allocation errors and reboots than =
the 2nd invocation which restricts itself to port 32000 and presumably just=
 one flow.=0A I wonder how to make cerowrt survive this kind of stress test=
=E2=80=A6 =0A=0Abest=0A Sebastian=0A=0A=0AOn Aug 15, 2012, at 9:08 PM, Dave=
 Taht wrote:=0A=0A> re: ath: skbuff alloc of size 1926 failed=0A> =0A> as f=
or the ath skbuff problem, I've seen that a lot. I had put hard=0A> packet =
limits (~600) on fq_codel in -11 and prior that were too low=0A> and it mos=
tly went away, but I hit tail drop behavior everywhere,=0A> instead of code=
l behavior. What I have now (typically 1200) may well=0A> be too high, but =
not as overly high as the default (10k packets).=0A> There may be another m=
eans of increasing the size of that slab pool or=0A> making it less onerous=
.=0A> =0A> I would like it if codel "kicked in" earlier than it currently d=
oes.=0A> The code in ns2 is currently using half the period that the linux =
code=0A> is. This would control things better, or so I hope (planning on tr=
ying=0A> this as I get time)=0A> =0A> I am also considering means of artifi=
cially upscaling the drop=0A> scheduler when we get close to queue limits.=
=0A> =0A> See some discussions on the codel list for these issues. (sims ar=
e=0A> easier to deal with than cerowrt, too!)=0A> =0A> as for bind, it shou=
ld be automagically restarted from xinetd, no need=0A> to fiddle with anyth=
ing. However, since you are already under massive=0A> memory pressure, it m=
ay well fail to start up that way, too. At the=0A> moment, I've largely giv=
en up on bind on anything but a more core home=0A> gw, and am running dnsma=
sq on everything (3700v2, picostations,=0A> nanostations) but the 3800s. (a=
nd the ones I run it on, aren't being=0A> used for wifi right now).=0A> =0A=
> Lastly: Swap space won't help you on exhausting kernel limits.=0A> =0A> I=
'm glad you can reproduce the ath: slab problem - I can get it too at=0A> h=
igh rates using netperf over wifi. I will try a 3700v2 with and=0A> without=
 bind to see if it's still there in 3.3.8-17. In the meantime=0A> if anyone=
 knows how to get more allocations in that (2048? 4096?) slab=0A> by defaul=
t, perhaps that will help?=0A> =0A> =0A> =0A> On Wed, Aug 15, 2012 at 10:23=
 AM, Sebastian Moeller <moeller0@gmx.de> wrote:=0A>> Hi Dave,=0A>> =0A>> gr=
eat work, as always I upgraded my production router to the latest and great=
est (since I only have one router=E2=80=A6). And it works quite well for no=
rmal usage=E2=80=A6=0A>> Netalyzr reports around 2800ms seconds of uplink b=
uffering, yet saturating the uplink does not affect ping times to a remote =
target noticeably, basically the same as for all codellized ceo versions I =
tested so far...=0A>> =0A>> Some notes and a question:=0A>> I noticed that =
even given plenty of swap space (1GB on a usb stick), using http://broadban=
d.mpi-sws.org/residential/ to exercise UDP stress (on the uplink I assume) =
I can easily produce (I run the test from a macosx via 5GHz wireless over 1=
.5 yards):=0A>> Aug 15 01:16:29 nacktmulle kern.err kernel: [175395.132812]=
 ath: skbuff alloc of size 1926 failed=0A>> (and plenty of those=E2=80=A6).=
=0A>> What then happens is that the OOM killer will aim for bind (reasonabl=
e since it is the largest single process) and kill it. When I try to restar=
t bind by:=0A>> root@nacktmulle:~# /etc/rc.d/S47namedprep start=0A>> root@n=
acktmulle:~# /etc/rc.d/S48named restart=0A>> Stopping isc-bind=0A>>  /etc/c=
hroot/named//var/run/named/named.pid not found, trying brute force=0A>> kil=
lall: named: no process killed=0A>> Kicking isc-bind in xinetd=0A>> rndc: c=
onnect failed: 127.0.0.1#953: connection refused=0A>> And bind does not sta=
rt again and the router becomes less than useful. Now I assume I am doing s=
omething wrong, but what, if you have any idea how to solve this short of a=
 reboot of the router (my current method) I would be happy to learn=0A>> =
=0A>> =0A>> =0A>> best regards=0A>>        sebastian=0A>> =0A>> On Aug 12, =
2012, at 11:08 PM, Dave Taht wrote:=0A>> =0A>>> I'm too tired to write up a=
 full set of release notes, but I've been=0A>>> testing it all day,=0A>>> a=
nd it looks better than -10 and certainly better than -11, but I won't know=
=0A>>> until some more folk sit down and test it, so here it is.=0A>>> =0A>=
>> http://huchra.bufferbloat.net/~cero1/3.3/3.3.8-17/=0A>>> =0A>>> fresh me=
rge with openwrt, fix to a bind CVE, fixes for 6in4 and quagga=0A>>> routin=
g problems,=0A>>> and a few tweaks to fq_codel setup that might make voip b=
etter.=0A>>> =0A>>> Go forth and break things!=0A>>> =0A>>> In other news:=
=0A>>> =0A>>> Van Jacobson gave a great talk about bufferbloat, BQL, codel,=
 and fq_codel=0A>>> at last week's ietf meeting. Well worth watching. At th=
e end he outlines=0A>>> the deployment problems in particular.=0A>>> =0A>>>=
 http://recordings.conf.meetecho.com/Recordings/watch.jsp?recording=3DIETF8=
4_TSVAREA&chapter=3Dpart_3=0A>>> =0A>>> Far more interesting than this emai=
l!=0A>>> =0A>>> =0A>>> --=0A>>> Dave T=C3=A4ht=0A>>> http://www.bufferbloat=
.net/projects/cerowrt/wiki - "3.3.8-17 is out=0A>>> with fq_codel!"=0A>>> _=
______________________________________________=0A>>> Cerowrt-devel mailing =
list=0A>>> Cerowrt-devel@lists.bufferbloat.net=0A>>> https://lists.bufferbl=
oat.net/listinfo/cerowrt-devel=0A>> =0A> =0A> =0A> =0A> -- =0A> Dave T=C3=
=A4ht=0A> http://www.bufferbloat.net/projects/cerowrt/wiki - "3.3.8-17 is o=
ut=0A> with fq_codel!"=0A=0A_______________________________________________=
=0ACerowrt-devel mailing list=0ACerowrt-devel@lists.bufferbloat.net=0Ahttps=
://lists.bufferbloat.net/listinfo/cerowrt-devel
------=_20120820223317000000_65240
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<font face=3D"arial" size=3D"2"><p style=3D"margin:0;padding:0;">I'm confus=
ed.&nbsp; Fq_codel does not (to my knowledge) fix bufferbloat *in your cabl=
e modem*&nbsp; or *in the CMTS head-end*.</p>=0A<p style=3D"margin:0;paddin=
g:0;">&nbsp;</p>=0A<p style=3D"margin:0;padding:0;">How could it?&nbsp;&nbs=
p; In order for that to be fixed, you need to manage the buffer in the cabl=
e modem itself.</p>=0A<p style=3D"margin:0;padding:0;">&nbsp;</p>=0A<p styl=
e=3D"margin:0;padding:0;">-----Original Message-----<br />From: "Sebastian =
Moeller" &lt;moeller0@gmx.de&gt;<br />Sent: Monday, August 20, 2012 2:24pm<=
br />To: "Dave Taht" &lt;dave.taht@gmail.com&gt;<br />Cc: cerowrt-devel@lis=
ts.bufferbloat.net<br />Subject: Re: [Cerowrt-devel] cerowrt 3.3.8-17 is re=
leased<br /><br /></p>=0A<div id=3D"SafeStyles1345516296">=0A<p style=3D"ma=
rgin:0;padding:0;">Hi Dave,<br /><br />so I went to play around with this a=
 bit more. I turned to UDP flooding my cable modem through the router and t=
his surely allows me to create enough load on the wndr3700v2 to cause the a=
llocation errors and as a "bonus" also to drive the router to reboot (drive=
n by the watchdog timer?). Here is the script I used over 5G wireless (from=
 http://blog.ioshints.info/2008/03/udp-flood-in-perl.html)<br /><br />#!/us=
r/bin/perl<br />##############<br /><br /># udp flood.<br />##############<=
br /><br />use Socket;<br />use strict;<br /><br />if ($#ARGV !=3D 3) {<br =
/> print "flood.pl &lt;ip&gt; &lt;port&gt; &lt;size&gt; &lt;time&gt;\n\n";<=
br /> print " port=3D0: use random ports\n";<br /> print " size=3D0: use ra=
ndom size between 64 and 1024\n";<br /> print " time=3D0: continuous flood\=
n";<br /> exit(1);<br />}<br /><br />my ($ip,$port,$size,$time) =3D @ARGV;<=
br /><br />my ($iaddr,$endtime,$psize,$pport);<br /><br />$iaddr =3D inet_a=
ton("$ip") or die "Cannot resolve hostname $ip\n";<br />$endtime =3D time()=
 + ($time ? $time : 1000000);<br /><br />socket(flood, PF_INET, SOCK_DGRAM,=
 17);<br /><br /><br />print "Flooding $ip " . ($port ? $port : "random") .=
 " port with " . <br /> ($size ? "$size-byte" : "random size") . " packets"=
 . <br /> ($time ? " for $time seconds" : "") . "\n";<br />print "Break wit=
h Ctrl-C\n" unless $time;<br /><br />for (;time() &lt;=3D $endtime;) {<br /=
> $psize =3D $size ? $size : int(rand(1024-64)+64) ;<br /> $pport =3D $port=
 ? $port : int(rand(65500))+1;<br /><br /> send(flood, pack("a$psize","floo=
d"), 0, pack_sockaddr_in($pport, $iaddr));}<br /><br />called as either<br =
/>udp_flood.pl 192.168.100.1 0 1024 240<br />or <br />udp_flood.pl 192.168.=
100.1 32000 1024 240<br /><br />The first version with randomized port numb=
er spreads the load nicely over many fq_codel bins/flows and seems slightly=
 more likely to cause allocation errors and reboots than the 2nd invocation=
 which restricts itself to port 32000 and presumably just one flow.<br /> I=
 wonder how to make cerowrt survive this kind of stress test=E2=80=A6 <br /=
><br />best<br /> Sebastian<br /><br /><br />On Aug 15, 2012, at 9:08 PM, D=
ave Taht wrote:<br /><br />&gt; re: ath: skbuff alloc of size 1926 failed<b=
r />&gt; <br />&gt; as for the ath skbuff problem, I've seen that a lot. I =
had put hard<br />&gt; packet limits (~600) on fq_codel in -11 and prior th=
at were too low<br />&gt; and it mostly went away, but I hit tail drop beha=
vior everywhere,<br />&gt; instead of codel behavior. What I have now (typi=
cally 1200) may well<br />&gt; be too high, but not as overly high as the d=
efault (10k packets).<br />&gt; There may be another means of increasing th=
e size of that slab pool or<br />&gt; making it less onerous.<br />&gt; <br=
 />&gt; I would like it if codel "kicked in" earlier than it currently does=
.<br />&gt; The code in ns2 is currently using half the period that the lin=
ux code<br />&gt; is. This would control things better, or so I hope (plann=
ing on trying<br />&gt; this as I get time)<br />&gt; <br />&gt; I am also =
considering means of artificially upscaling the drop<br />&gt; scheduler wh=
en we get close to queue limits.<br />&gt; <br />&gt; See some discussions =
on the codel list for these issues. (sims are<br />&gt; easier to deal with=
 than cerowrt, too!)<br />&gt; <br />&gt; as for bind, it should be automag=
ically restarted from xinetd, no need<br />&gt; to fiddle with anything. Ho=
wever, since you are already under massive<br />&gt; memory pressure, it ma=
y well fail to start up that way, too. At the<br />&gt; moment, I've largel=
y given up on bind on anything but a more core home<br />&gt; gw, and am ru=
nning dnsmasq on everything (3700v2, picostations,<br />&gt; nanostations) =
but the 3800s. (and the ones I run it on, aren't being<br />&gt; used for w=
ifi right now).<br />&gt; <br />&gt; Lastly: Swap space won't help you on e=
xhausting kernel limits.<br />&gt; <br />&gt; I'm glad you can reproduce th=
e ath: slab problem - I can get it too at<br />&gt; high rates using netper=
f over wifi. I will try a 3700v2 with and<br />&gt; without bind to see if =
it's still there in 3.3.8-17. In the meantime<br />&gt; if anyone knows how=
 to get more allocations in that (2048? 4096?) slab<br />&gt; by default, p=
erhaps that will help?<br />&gt; <br />&gt; <br />&gt; <br />&gt; On Wed, A=
ug 15, 2012 at 10:23 AM, Sebastian Moeller &lt;moeller0@gmx.de&gt; wrote:<b=
r />&gt;&gt; Hi Dave,<br />&gt;&gt; <br />&gt;&gt; great work, as always I =
upgraded my production router to the latest and greatest (since I only have=
 one router=E2=80=A6). And it works quite well for normal usage=E2=80=A6<br=
 />&gt;&gt; Netalyzr reports around 2800ms seconds of uplink buffering, yet=
 saturating the uplink does not affect ping times to a remote target notice=
ably, basically the same as for all codellized ceo versions I tested so far=
...<br />&gt;&gt; <br />&gt;&gt; Some notes and a question:<br />&gt;&gt; I=
 noticed that even given plenty of swap space (1GB on a usb stick), using h=
ttp://broadband.mpi-sws.org/residential/ to exercise UDP stress (on the upl=
ink I assume) I can easily produce (I run the test from a macosx via 5GHz w=
ireless over 1.5 yards):<br />&gt;&gt; Aug 15 01:16:29 nacktmulle kern.err =
kernel: [175395.132812] ath: skbuff alloc of size 1926 failed<br />&gt;&gt;=
 (and plenty of those=E2=80=A6).<br />&gt;&gt; What then happens is that th=
e OOM killer will aim for bind (reasonable since it is the largest single p=
rocess) and kill it. When I try to restart bind by:<br />&gt;&gt; root@nack=
tmulle:~# /etc/rc.d/S47namedprep start<br />&gt;&gt; root@nacktmulle:~# /et=
c/rc.d/S48named restart<br />&gt;&gt; Stopping isc-bind<br />&gt;&gt;  /etc=
/chroot/named//var/run/named/named.pid not found, trying brute force<br />&=
gt;&gt; killall: named: no process killed<br />&gt;&gt; Kicking isc-bind in=
 xinetd<br />&gt;&gt; rndc: connect failed: 127.0.0.1#953: connection refus=
ed<br />&gt;&gt; And bind does not start again and the router becomes less =
than useful. Now I assume I am doing something wrong, but what, if you have=
 any idea how to solve this short of a reboot of the router (my current met=
hod) I would be happy to learn<br />&gt;&gt; <br />&gt;&gt; <br />&gt;&gt; =
<br />&gt;&gt; best regards<br />&gt;&gt;        sebastian<br />&gt;&gt; <b=
r />&gt;&gt; On Aug 12, 2012, at 11:08 PM, Dave Taht wrote:<br />&gt;&gt; <=
br />&gt;&gt;&gt; I'm too tired to write up a full set of release notes, bu=
t I've been<br />&gt;&gt;&gt; testing it all day,<br />&gt;&gt;&gt; and it =
looks better than -10 and certainly better than -11, but I won't know<br />=
&gt;&gt;&gt; until some more folk sit down and test it, so here it is.<br /=
>&gt;&gt;&gt; <br />&gt;&gt;&gt; http://huchra.bufferbloat.net/~cero1/3.3/3=
.3.8-17/<br />&gt;&gt;&gt; <br />&gt;&gt;&gt; fresh merge with openwrt, fix=
 to a bind CVE, fixes for 6in4 and quagga<br />&gt;&gt;&gt; routing problem=
s,<br />&gt;&gt;&gt; and a few tweaks to fq_codel setup that might make voi=
p better.<br />&gt;&gt;&gt; <br />&gt;&gt;&gt; Go forth and break things!<b=
r />&gt;&gt;&gt; <br />&gt;&gt;&gt; In other news:<br />&gt;&gt;&gt; <br />=
&gt;&gt;&gt; Van Jacobson gave a great talk about bufferbloat, BQL, codel, =
and fq_codel<br />&gt;&gt;&gt; at last week's ietf meeting. Well worth watc=
hing. At the end he outlines<br />&gt;&gt;&gt; the deployment problems in p=
articular.<br />&gt;&gt;&gt; <br />&gt;&gt;&gt; http://recordings.conf.meet=
echo.com/Recordings/watch.jsp?recording=3DIETF84_TSVAREA&amp;chapter=3Dpart=
_3<br />&gt;&gt;&gt; <br />&gt;&gt;&gt; Far more interesting than this emai=
l!<br />&gt;&gt;&gt; <br />&gt;&gt;&gt; <br />&gt;&gt;&gt; --<br />&gt;&gt;=
&gt; Dave T=C3=A4ht<br />&gt;&gt;&gt; http://www.bufferbloat.net/projects/c=
erowrt/wiki - "3.3.8-17 is out<br />&gt;&gt;&gt; with fq_codel!"<br />&gt;&=
gt;&gt; _______________________________________________<br />&gt;&gt;&gt; C=
erowrt-devel mailing list<br />&gt;&gt;&gt; Cerowrt-devel@lists.bufferbloat=
.net<br />&gt;&gt;&gt; https://lists.bufferbloat.net/listinfo/cerowrt-devel=
<br />&gt;&gt; <br />&gt; <br />&gt; <br />&gt; <br />&gt; -- <br />&gt; Da=
ve T=C3=A4ht<br />&gt; http://www.bufferbloat.net/projects/cerowrt/wiki - "=
3.3.8-17 is out<br />&gt; with fq_codel!"<br /><br />______________________=
_________________________<br />Cerowrt-devel mailing list<br />Cerowrt-deve=
l@lists.bufferbloat.net<br />https://lists.bufferbloat.net/listinfo/cerowrt=
-devel</p>=0A</div></font>
------=_20120820223317000000_65240--