From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-13-ewr.dyndns.com (mxout-050-ewr.mailhop.org [216.146.33.50]) by lists.bufferbloat.net (Postfix) with ESMTP id 6D7A12E060A for ; Mon, 18 Apr 2011 05:18:21 -0700 (PDT) Received: from scan-12-ewr.mailhop.org (scan-12-ewr.local [10.0.141.230]) by mail-13-ewr.dyndns.com (Postfix) with ESMTP id D169CA51491 for ; Mon, 18 Apr 2011 12:18:20 +0000 (UTC) X-Spam-Score: 1.5 (+) X-Mail-Handler: MailHop by DynDNS X-Originating-IP: 149.20.54.64 Received: from mainmail.teklibre.com (toutatis.isc.org [149.20.54.64]) by mail-13-ewr.dyndns.com (Postfix) with ESMTP id 1B8D2A5139F for ; Mon, 18 Apr 2011 12:18:15 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mainmail.teklibre.com (Postfix) with ESMTP id CA4553DA139; Mon, 18 Apr 2011 05:52:47 -0700 (PDT) Received: from mainmail.teklibre.com ([127.0.0.1]) by localhost (toutatis.sql1.isc.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id z28lF-RCmFJe; Mon, 18 Apr 2011 05:52:47 -0700 (PDT) Received: from [192.168.1.4] (c-68-39-173-20.hsd1.nj.comcast.net [68.39.173.20]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) (Authenticated sender: d) by mainmail.teklibre.com (Postfix) with ESMTPSA id C8F613DA121; Mon, 18 Apr 2011 05:52:46 -0700 (PDT) Message-ID: <4DAC2C03.2060502@taht.net> Date: Mon, 18 Apr 2011 06:18:11 -0600 From: Dave Taht User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.14) Gecko/20110223 Thunderbird/3.1.8 MIME-Version: 1.0 To: hndr@lists.pixilab.org, bismark-devel@lists.bufferbloat.net, abhishekjain95@gmail.com References: <27196F99-CB9E-4963-8AA9-71A66B567B69@cc.gatech.edu> In-Reply-To: <27196F99-CB9E-4963-8AA9-71A66B567B69@cc.gatech.edu> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [Bismark-devel] [Hndr] FCC app progress: django X-BeenThere: bismark-devel@lists.bufferbloat.net X-Mailman-Version: 2.1.13 Precedence: list List-Id: BISMark related software development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 18 Apr 2011 12:18:21 -0000 On 04/17/2011 11:57 PM, Nick Feamster wrote: > (sorry for list cross-posting) > > Srikanth, Abhishek, Walter, Taking walter and sri off the explicit cc list as both are already on bismark-devel. > Small, slow progress on the FCC app--- > > I have a written a very simple server with django. I've got the django server hooked up to our bismark mysql database. It's basically a slick python wrapper for doing Web development with a database back-end. The setup is ugly right now, but it's a proof of concept, and it works (django is talking to our back-end). For example: > http://networkdashboard.org:8000/summary/ > How we have configured the web servers here has had us survive several slashdottings. http://www.bufferbloat.net/projects/bloat/wiki/Dogfood_Principle Using fast cgi vs wsgi would probably lead to some security benefits and more predictable behavior, however it's a tossup. http://docs.djangoproject.com/en/dev/howto/deployment/fastcgi/ > Django has templates that will allow us to make this slick and pretty, but it's getting a little late for that tonight. > > I think we can whip up some quick visualizations of what's in the DB with a nice Web page with some concerted python hacking. For starters, it would be nice to have a version of the data whereby queries can actually complete in a reasonable time. :-) What's the best way to do this? Split the table by months? > Your mysql configuration was an out of the box default one designed for very small databases. Yours is quite large. Increasing several buffer sizes by several orders of magnitude would help a lot. Sri had made significant inroads into porting the database into postgres when I was there. Postgres is not only more highly performant and flexible than mysql, with a more capable parser, but has good analysis tools. It was my hope to benchmark some of your slowest existing queries using postgres's ANALYZE tool to see how the design could be improved overall. I did look over one of the slower (30+ seconds!) queries and suggested that a between statement be used instead of an and for the date range, but we never got around to testing that and mysql used to not be able to do anything sane with a between range (postgres can). e.g: mysql> select dstport, sum(uppkts) as up_p, sum(dwpkts) as down_p, sum(upbytes) as up_bytes, sum(dwbytes) as down_bytes from FLOWS_newformat where tsstart>unix_timestamp('2011-2-6') and tsstart I will check what I have into the bismark source tree so that others can work on this. (I'm looking at Abhishek :-) > While not exactly allergic to svn, I am a git fan-boy. Even when working with svn these days I use git-svn to cope with it (offline git logs rule). I'm not asking you to change your scm backend, just sayin...