Monday, July 25, 2011

Zarkov is a Web Service

Over the past few weeks I've been working on a service in Python that I'm calling, in the tradition of naming projects after characters in Flash Gordon, Zarkov. So what exactly is Zarkov? Well, Zarkov is many things (and may grow to more):

In my first post, I discussed Zarkov as an event logger. Next, I explored Zarkov as a map-reduce framework. After that, I talked about using Zarkov's aggregation definitions. Today, I'll focus on how you can get those aggregates out of Zarkov using Zarkov's built-in JSON web service.


Basic usage

In order to expose Zarkov's webservice, you'll need to add a key to your development.yaml telling it what port to listen on:

web_port: 8081

Once that's set up, you need to start the Zarkov server:

(zarkov) $ zcmd -y development.yaml serve

Now we need some data to be put into Zarkov. Luckily, Zarkov comes with a 'perftest' command that we can use to fill the data:

(zarkov) $ zcmd -y development.yaml init
(zarkov) $ # zcmd perftest <url> <num_events> <concurrency> <rate> 
(zarkov) $ zcmd -y development.yaml perftest tcp://localhost:6543 1000 1 100
(zarkov) $ zcmd -y development.yaml aggregate perftest

Now you can request data from Zarkov by using simple HTML form requests, receiving the results in JSON:

(zarkov) $ curl 'http://127.0.0.1:8081/q?p=perftest.&c=sum_by_second' 
{"v": [["00", 100], ["01", 101], ["02", 101], ["03", 101], ["04", 101], ["05", 101], ["06", 101], ["07", 101], ["08", 101]]} 
(zarkov) $ curl 'http://127.0.0.1:8081/q?p=perftest.&c=sum_by_second&b=03&e=07' 
{"v": [["03", 101], ["04", 101], ["05", 101], ["06", 101], ["07", 101]]}

Integrating with Flot

The data format returned by the Zarkov web service is a list of (key, value) pairs for every value calculated in the aggregate. If you compute your aggregates using ISO timestamps as part of the key, Zarkov will helpfully convert these timestamps to Flot-acceptable timestamps for graphing:

(zarkov) $ curl 'http://127.0.0.1:8081/q?p=perftest.&c=sum_by_second_ts' 
{"v": [[1311626220000.0, 100], [1311626221000.0, 101], [1311626222000.0, 101], [1311626223000.0, 101], [1311626224000.0, 101], [1311626225000.0, 101], [1311626226000.0, 101], [1311626227000.0, 101], [1311626228000.0, 101]]}

In order to use the automatic timestamp feature, you'll need to make sure that your key ends with an ISO formatted datetime prefixed by a slash ("/") character. You can set the definition of the perftest task in zarkov.helpers to see how this can be done. Also note that you can specify a prefix argument to your query. This is prepended onto the collection name. If it's not provided, Zarkov will assume you mean the zagg. prefix.

To demonstrate the flot integration, let's set up a tiny webapp in /tmp/zarkov. First we'll create the directory structure and grab a copy of flot:

(zarkov) $ mkdir /tmp/zarkov
(zarkov) $ mkdir /tmp/zarkov/static
(zarkov) $ pushd /tmp/zarkov/static
(zarkov) $ wget http://flot.googlecode.com/files/flot-0.7.tar.gz
(zarkov) $ tar xzf flot-0.7.tar.gz

Now we'll create an index page in /tmp/zarkov/index.html:

<html> 
  <head> 
    <title>Zarkov Graph Test</title> 
  </head> 
   <div id="placeholder" style="width:600px;height:300px"></div> 
    <script type="text/javascript" src="flot/jquery.min.js"></script> 
    <script type="text/javascript" src="flot/jquery.flot.min.js"></script> 
  <!--[if lte IE 8]><script language="javascript" type="text/javascript" src="/static/flot/excanvas.min.js"></script><![endif]--> 
  <script type="text/javascript" src="index.js"></script> 
  </body> 
</html>

and the corresponding index.js:

(function() { 
    var $placeholder = $('div#placeholder'); 
    var plot_options = { 
        xaxes: [ {mode:'time' }]}; 
    var params = { 
        p: "perftest.", c:"sum_by_second_ts" }; 
    $.getJSON( 
        '/ws/q', 
        $.param(params), 
        function(result) { 
            var series = []; 
            $.each(result, function(key, value) { 
                series.push( {label:key, data:value } ); 
            }); 
            $.plot($placeholder, series, plot_options); 
        }); 
}());

And here's our little webapp. You'll need to pip install paste if you don't already have it.

from gevent.pywsgi import WSGIServer 
from paste.urlparser import StaticURLParser 
from paste.urlmap import URLMap 
from zarkov.web import WebApp 
 
app = URLMap() 
app['/static'] = StaticURLParser('/tmp/zarkov/static') 
app['/ws'] = WebApp() 
 
web_server = WSGIServer(('', 8081), app) 
web_server.serve_forever()

Now if we run the webapp with the Zarkov script command and visit http://localhost:8081/static/index.html, we can see a pretty graph:

(zarkov) $ zcmd -y development.yaml script /tmp/zarkov/test.py


Hope you've enjoyed this brief intro to Zarkov. If you've got any questions, comments, or suggestions please let me know either in the comments, Zarkov's discussion forums, or Zarkov's ticket tracker (all conveniently located at http://sf.net/p/zarkov). Happy hacking!

Posted by Rick Copeland at 12:51 PM
Labels: , , , , ,

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)