From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <moeller0@gmx.de>
Received: from mailout-de.gmx.net (mailout-de.gmx.net [213.165.64.22])
	by huchra.bufferbloat.net (Postfix) with SMTP id 6ED9F21F0E6
	for <cerowrt-devel@lists.bufferbloat.net>;
	Mon, 20 Aug 2012 22:28:25 -0700 (PDT)
Received: (qmail invoked by alias); 21 Aug 2012 05:28:23 -0000
Received: from 75-142-58-156.static.mtpk.ca.charter.com (EHLO
	dhcp-112.home.lan) [75.142.58.156]
	by mail.gmx.net (mp030) with SMTP; 21 Aug 2012 07:28:23 +0200
X-Authenticated: #24211782
X-Provags-ID: V01U2FsdGVkX18pApU3AqwpSnjK/s3UdFyaJMoiOEILk1bPnQ/PW5
	XSPC4AKyTo2cUv
Mime-Version: 1.0 (Apple Message framework v1278)
Content-Type: text/plain; charset=windows-1252
From: Sebastian Moeller <moeller0@gmx.de>
In-Reply-To: <E7CFED87-2E10-4DF9-8A6B-06B2787476A1@gmail.com>
Date: Mon, 20 Aug 2012 22:28:20 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <B864DF8C-45AD-464E-8732-C2F6B5ED96C4@gmx.de>
References: <CAA93jw50MeqWH6TVKditFGfg-V-mOi-UUtsABqd+WHs2vedHQw@mail.gmail.com>
	<36D61FDC-9AA9-46CC-ACBB-2D28B250C660@gmx.de>
	<CAA93jw5JV0L0MLYxuO-ok2joOkodie_JkxNs_VAJat_eYhV1gw@mail.gmail.com>
	<2998C331-777B-41B3-A6BC-8285461EF729@gmx.de>
	<1345516397.664231592@apps.rackspace.com>
	<E7CFED87-2E10-4DF9-8A6B-06B2787476A1@gmail.com>
To: Marchon <marchon@gmail.com>
X-Mailer: Apple Mail (2.1278)
X-Y-GMX-Trusted: 0
Cc: "cerowrt-devel@lists.bufferbloat.net" <cerowrt-devel@lists.bufferbloat.net>
Subject: Re: [Cerowrt-devel] cerowrt 3.3.8-17 is released
X-BeenThere: cerowrt-devel@lists.bufferbloat.net
X-Mailman-Version: 2.1.13
Precedence: list
List-Id: Development issues regarding the cerowrt test router project
	<cerowrt-devel.lists.bufferbloat.net>
List-Unsubscribe: <https://lists.bufferbloat.net/options/cerowrt-devel>,
	<mailto:cerowrt-devel-request@lists.bufferbloat.net?subject=unsubscribe>
List-Archive: <https://lists.bufferbloat.net/pipermail/cerowrt-devel>
List-Post: <mailto:cerowrt-devel@lists.bufferbloat.net>
List-Help: <mailto:cerowrt-devel-request@lists.bufferbloat.net?subject=help>
List-Subscribe: <https://lists.bufferbloat.net/listinfo/cerowrt-devel>,
	<mailto:cerowrt-devel-request@lists.bufferbloat.net?subject=subscribe>
X-List-Received-Date: Tue, 21 Aug 2012 05:28:26 -0000

Hi there,

again the UDP flood test was not about the cable modem but just about =
resilience under (extreme) load. So just plain bread and butter =
functionality of a router, nothing fancy :) As is it causes a mix of =
very slow processing due to allocation errors and occasional OOM =
situations and full reboots. IMHO surviving the stress without these =
unfortunate outcomes would be preferable=85


best
	Sebastian

On Aug 20, 2012, at 7:44 PM, Marchon wrote:

> It The Fq_codel settings prevent excess buffering on the push of data =
out over the cable modem itself / it will prevent unnecessary premature =
reduction of the  tcp sliding window size further preventing a cascading =
backlog that ends up further reducing the sliding window and slowing =
down the overall outbound transfer rate.=20
>=20
> A buffering problem in the cable modem only happens if you feed it =
data to quickly.=20
>=20
>=20
>=20
>=20
>=20
>=20
> Sent from my iPhone
>=20
> On Aug 20, 2012, at 10:33 PM, dpreed@reed.com wrote:
>=20
>> I'm confused.  Fq_codel does not (to my knowledge) fix bufferbloat =
*in your cable modem*  or *in the CMTS head-end*.
>> =20
>> How could it?   In order for that to be fixed, you need to manage the =
buffer in the cable modem itself.
>> =20
>> -----Original Message-----
>> From: "Sebastian Moeller" <moeller0@gmx.de>
>> Sent: Monday, August 20, 2012 2:24pm
>> To: "Dave Taht" <dave.taht@gmail.com>
>> Cc: cerowrt-devel@lists.bufferbloat.net
>> Subject: Re: [Cerowrt-devel] cerowrt 3.3.8-17 is released
>>=20
>> Hi Dave,
>>=20
>> so I went to play around with this a bit more. I turned to UDP =
flooding my cable modem through the router and this surely allows me to =
create enough load on the wndr3700v2 to cause the allocation errors and =
as a "bonus" also to drive the router to reboot (driven by the watchdog =
timer?). Here is the script I used over 5G wireless (from =
http://blog.ioshints.info/2008/03/udp-flood-in-perl.html)
>>=20
>> #!/usr/bin/perl
>> ##############
>>=20
>> # udp flood.
>> ##############
>>=20
>> use Socket;
>> use strict;
>>=20
>> if ($#ARGV !=3D 3) {
>> print "flood.pl <ip> <port> <size> <time>\n\n";
>> print " port=3D0: use random ports\n";
>> print " size=3D0: use random size between 64 and 1024\n";
>> print " time=3D0: continuous flood\n";
>> exit(1);
>> }
>>=20
>> my ($ip,$port,$size,$time) =3D @ARGV;
>>=20
>> my ($iaddr,$endtime,$psize,$pport);
>>=20
>> $iaddr =3D inet_aton("$ip") or die "Cannot resolve hostname $ip\n";
>> $endtime =3D time() + ($time ? $time : 1000000);
>>=20
>> socket(flood, PF_INET, SOCK_DGRAM, 17);
>>=20
>>=20
>> print "Flooding $ip " . ($port ? $port : "random") . " port with " .=20=

>> ($size ? "$size-byte" : "random size") . " packets" .=20
>> ($time ? " for $time seconds" : "") . "\n";
>> print "Break with Ctrl-C\n" unless $time;
>>=20
>> for (;time() <=3D $endtime;) {
>> $psize =3D $size ? $size : int(rand(1024-64)+64) ;
>> $pport =3D $port ? $port : int(rand(65500))+1;
>>=20
>> send(flood, pack("a$psize","flood"), 0, pack_sockaddr_in($pport, =
$iaddr));}
>>=20
>> called as either
>> udp_flood.pl 192.168.100.1 0 1024 240
>> or=20
>> udp_flood.pl 192.168.100.1 32000 1024 240
>>=20
>> The first version with randomized port number spreads the load nicely =
over many fq_codel bins/flows and seems slightly more likely to cause =
allocation errors and reboots than the 2nd invocation which restricts =
itself to port 32000 and presumably just one flow.
>> I wonder how to make cerowrt survive this kind of stress test=85=20
>>=20
>> best
>> Sebastian
>>=20
>>=20
>> On Aug 15, 2012, at 9:08 PM, Dave Taht wrote:
>>=20
>> > re: ath: skbuff alloc of size 1926 failed
>> >=20
>> > as for the ath skbuff problem, I've seen that a lot. I had put hard
>> > packet limits (~600) on fq_codel in -11 and prior that were too low
>> > and it mostly went away, but I hit tail drop behavior everywhere,
>> > instead of codel behavior. What I have now (typically 1200) may =
well
>> > be too high, but not as overly high as the default (10k packets).
>> > There may be another means of increasing the size of that slab pool =
or
>> > making it less onerous.
>> >=20
>> > I would like it if codel "kicked in" earlier than it currently =
does.
>> > The code in ns2 is currently using half the period that the linux =
code
>> > is. This would control things better, or so I hope (planning on =
trying
>> > this as I get time)
>> >=20
>> > I am also considering means of artificially upscaling the drop
>> > scheduler when we get close to queue limits.
>> >=20
>> > See some discussions on the codel list for these issues. (sims are
>> > easier to deal with than cerowrt, too!)
>> >=20
>> > as for bind, it should be automagically restarted from xinetd, no =
need
>> > to fiddle with anything. However, since you are already under =
massive
>> > memory pressure, it may well fail to start up that way, too. At the
>> > moment, I've largely given up on bind on anything but a more core =
home
>> > gw, and am running dnsmasq on everything (3700v2, picostations,
>> > nanostations) but the 3800s. (and the ones I run it on, aren't =
being
>> > used for wifi right now).
>> >=20
>> > Lastly: Swap space won't help you on exhausting kernel limits.
>> >=20
>> > I'm glad you can reproduce the ath: slab problem - I can get it too =
at
>> > high rates using netperf over wifi. I will try a 3700v2 with and
>> > without bind to see if it's still there in 3.3.8-17. In the =
meantime
>> > if anyone knows how to get more allocations in that (2048? 4096?) =
slab
>> > by default, perhaps that will help?
>> >=20
>> >=20
>> >=20
>> > On Wed, Aug 15, 2012 at 10:23 AM, Sebastian Moeller =
<moeller0@gmx.de> wrote:
>> >> Hi Dave,
>> >>=20
>> >> great work, as always I upgraded my production router to the =
latest and greatest (since I only have one router=85). And it works =
quite well for normal usage=85
>> >> Netalyzr reports around 2800ms seconds of uplink buffering, yet =
saturating the uplink does not affect ping times to a remote target =
noticeably, basically the same as for all codellized ceo versions I =
tested so far...
>> >>=20
>> >> Some notes and a question:
>> >> I noticed that even given plenty of swap space (1GB on a usb =
stick), using http://broadband.mpi-sws.org/residential/ to exercise UDP =
stress (on the uplink I assume) I can easily produce (I run the test =
from a macosx via 5GHz wireless over 1.5 yards):
>> >> Aug 15 01:16:29 nacktmulle kern.err kernel: [175395.132812] ath: =
skbuff alloc of size 1926 failed
>> >> (and plenty of those=85).
>> >> What then happens is that the OOM killer will aim for bind =
(reasonable since it is the largest single process) and kill it. When I =
try to restart bind by:
>> >> root@nacktmulle:~# /etc/rc.d/S47namedprep start
>> >> root@nacktmulle:~# /etc/rc.d/S48named restart
>> >> Stopping isc-bind
>> >> /etc/chroot/named//var/run/named/named.pid not found, trying brute =
force
>> >> killall: named: no process killed
>> >> Kicking isc-bind in xinetd
>> >> rndc: connect failed: 127.0.0.1#953: connection refused
>> >> And bind does not start again and the router becomes less than =
useful. Now I assume I am doing something wrong, but what, if you have =
any idea how to solve this short of a reboot of the router (my current =
method) I would be happy to learn
>> >>=20
>> >>=20
>> >>=20
>> >> best regards
>> >> sebastian
>> >>=20
>> >> On Aug 12, 2012, at 11:08 PM, Dave Taht wrote:
>> >>=20
>> >>> I'm too tired to write up a full set of release notes, but I've =
been
>> >>> testing it all day,
>> >>> and it looks better than -10 and certainly better than -11, but I =
won't know
>> >>> until some more folk sit down and test it, so here it is.
>> >>>=20
>> >>> http://huchra.bufferbloat.net/~cero1/3.3/3.3.8-17/
>> >>>=20
>> >>> fresh merge with openwrt, fix to a bind CVE, fixes for 6in4 and =
quagga
>> >>> routing problems,
>> >>> and a few tweaks to fq_codel setup that might make voip better.
>> >>>=20
>> >>> Go forth and break things!
>> >>>=20
>> >>> In other news:
>> >>>=20
>> >>> Van Jacobson gave a great talk about bufferbloat, BQL, codel, and =
fq_codel
>> >>> at last week's ietf meeting. Well worth watching. At the end he =
outlines
>> >>> the deployment problems in particular.
>> >>>=20
>> >>> =
http://recordings.conf.meetecho.com/Recordings/watch.jsp?recording=3DIETF8=
4_TSVAREA&chapter=3Dpart_3
>> >>>=20
>> >>> Far more interesting than this email!
>> >>>=20
>> >>>=20
>> >>> --
>> >>> Dave T=E4ht
>> >>> http://www.bufferbloat.net/projects/cerowrt/wiki - "3.3.8-17 is =
out
>> >>> with fq_codel!"
>> >>> _______________________________________________
>> >>> Cerowrt-devel mailing list
>> >>> Cerowrt-devel@lists.bufferbloat.net
>> >>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>> >>=20
>> >=20
>> >=20
>> >=20
>> > --=20
>> > Dave T=E4ht
>> > http://www.bufferbloat.net/projects/cerowrt/wiki - "3.3.8-17 is out
>> > with fq_codel!"
>>=20
>> _______________________________________________
>> Cerowrt-devel mailing list
>> Cerowrt-devel@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cerowrt-devel
>> _______________________________________________
>> Cerowrt-devel mailing list
>> Cerowrt-devel@lists.bufferbloat.net
>> https://lists.bufferbloat.net/listinfo/cerowrt-devel