Lazy Descriptors

Today I had a need to create a property on an object "lazily." The Python builtin property does a great job of this, but it calls the getter function every time you access the property. Here is how I ended up solving the problem:

First of all, I had (almost) the behavior I wanted by using the following pattern:

class Foo(object):
    def __init__(self):
        self._bar = None
    @property
    def bar(self):
        if self._bar is None:
            print 'Calculating self._bar'  
            self._bar = 42    
        return self._bar      

There are a couple of problems with this, however. First of all, I'm polluting my object's namespace with a _bar attribute that I don't want. Secondly, I'm using this pattern all over my codebase, and it's quite an eyesore.

Both problems can be fixed by using a descriptor. Basically, a descriptor is an object with a __get__ method which is called when the descriptor is accessed as a property of a class. The descriptor I created is below:

class LazyProperty(object):

    def __init__(self, func):
        self._func = func
        self.__name__ = func.__name__
        self.__doc__ = func.__doc__

    def __get__(self, obj, klass=None):
        if obj is None: return None
        result = obj.__dict__[self.__name__] = self._func(obj)
        return result

The descriptor is designed to be used as a decorator, and will save the decorated function and its name. When the descriptor is accessed, it will calculate the value by calling the function and save the calculated value back to the object's dict. Saving back to the object's dict has the additional benefit of preventing the descriptor from being called the next time the property is accessed. So I can now use it in the class above:

class Foo(object):
    @LazyProperty
    def bar(self):
        print 'Calculating self._bar'  
        return 42

So I get a nice lazily calculated property that doesn't recalculate bar every time it's accessed and doesn't bother with any memoization itself. What do you think about it? Is this a patten you use in your code?

New Domain blog.pythonisito.com

I just wanted to let you all know that I've changed from the blogger domain to my own blog.pythonisito.com. You should be redirected there automatically, but if you've noticed some hiccups in feeds or weird redirects from Reddit or Delicious, now you know why.

A Little Command Line Love

One of the things I do in my "spare" time is work on building web applications that will (hopefully) earn some spare money on the side without too much maintenance on my part. Those who have read The Four Hour Work Week will recognize this as my "muse" business.

In working on these web apps, I needed a place to host them, so I went with WebFaction due to their excellent support for TurboGears. I'm using a shared hosting environment with WebFaction, so my usual method of getting stuff up can't involve any scripts in /etc/init.d like I'd use at work, so I used to do the old nohup python start-appname.py >> output.log 2>&1 & to "daemonize" the process and then ps -furick446 to figure out which processes I needed to kill/restart when updating code. This was irritatingly verbose, so I figured I'd build a "userspace daemonizer" which I'll describe below.

The requirements of my daemonizer were pretty straightforward:

• I should be able to add/remove services from the command line with a minimum amount of typing


• I should be able to start/stop/restart any service just by listing it by name (no more ps -furick446)


• It should do "real" daemonization (fork, fork, dup stuff if you're familiar with it) rather than the nohup garbage I'd been using


• It should perform some sort of verification or status check on processes to make sure that they're successfully started/stopped/running/etc.

The first (and most questionable) decision I made was to go with SQLAlchemy as my service database. I say this is questionable because I ended up with only one table and one client that uses the database, so a fully relational model is really overkill here. Anyway, here's the SQLAlchemy setup code I used. It's pretty simple, and represents kind of the "lowest common denominator" of SQLAlchemy usage. (I've also included all the imports required for the whole shebang at the top of the file.)

#!/usr/bin/env python2.5
import os, sys, signal, shlex, time
from optparse import OptionParser

from sqlalchemy import *
from sqlalchemy.orm import *

HOME=os.environ.get('HOME')
DBFILE=os.path.abspath(os.environ.get(
    'SERVER_FILE',
    os.path.join(HOME, 'server.sqlite')))

DBURI='sqlite:///' + DBFILE

engine = create_engine(DBURI)
metadata = MetaData(engine)
session = scoped_session(sessionmaker(bind=engine))

process = Table(
    'process', metadata,
    Column('name', String(255), primary_key=True),
    Column('pid', Integer),
    Column('command_line', String(255)),
    Column('working_directory', String(255)),
    Column('stdin', String(255), default='/dev/null'),
    Column('stdout', String(255), default='/dev/null'),
    Column('stderr', String(255), default='/dev/null'))

class Process(object):
    def __repr__(self):
        return '%s(PID %s, WD %s): %s < %s >> %s 2>> %s' % (
            self.name, self.pid, self.working_directory,
            self.command_line, self.stdin, self.stdout, self.stderr)
session.mapper(Process, process)      

So, for all of you out there who wonder how to use SQLAlchemy outside of a web framework, there you go. The idea here is that every service has a row in the process table, and every service can optionally redirect its stdin/stdout/stderr to/from files. If a process is running, it will have a pid. You can also specify the startup directory of each process. That pretty much covers the data model.

My next task was to figure out how to invoke this from the command line. I decided to make all the commands of the form python server.py command [options] [service]. Actually, the only command that takes any options is the add command, so I definitely over-engineered here, but I wanted to be able to specify a global list of options shared by all commands to be used with an optparse option parser. So I built the following dictionary:

OPTPARSE_OPTIONS = {
    'working-directory':(['-w', '--working-directory'],
                         dict(dest='ws', default=HOME,
                              help='Set working directory to DIR',
                              metavar='DIR')),
    'stdin':(['-i', '--stdin'],
             dict(dest='stdin', default='/dev/null',
                  help='Use FILE as stdin', metavar='FILE')),
    'stdout':(['-o', '--stdout'],
             dict(dest='stdout', default='/dev/null',
                  help='Use FILE as stdout', metavar='FILE')),
    'stderr':(['-e', '--stderr'],
             dict(dest='stderr', default='/dev/null',
                  help='Use FILE as stderr', metavar='FILE')),
    }      

The next task was to specify the commands. I'm lazy and I like decorators, so I decided that new commands should be as easy as possible to write. My add command, for instance, is the most complex, and it looks like this:

@Command
def add(service, command, working_directory,
        stdin, stdout, stderr):
    p = Process(name=service,
                command_line=command,
                working_directory=working_directory,
                stdin=stdin,
                stdout=stdout,
                stderr=stderr)
    session.commit()      

That's pretty simple. Of course, as you might have guessed, the Command decorator isn't all that simple. Its responsibilities are as follows:

• Add the function name to a command list so I can get a list of commands from the command line


• Build an optparse parser based on the OPTPARSE_OPTIONS dict and the named arguments to the function


• Wrap the function in a new function with a signature like function(args) so it can be called with sys.argv[2:]


• In the wrapper function, use the optparse parser to initialize the argument list to the function and then call with a named argument dict
  

So with all that explanation, here's the code:

class Command(object):
    commands = {}
    def __init__(self, func):
        Command.commands[func.__name__] = self
        options = self.get_options(func)
        self.func = func
        self.optparse_options = []
        self.optparse_option_names = []
        self.positional_options = []
        for o in options:
            if o in OPTPARSE_OPTIONS:
                self.optparse_option_names.append(o)
                self.optparse_options.append(
                    OPTPARSE_OPTIONS[o])
            else:
                self.positional_options.append(o)
        positional_string = ' '.join(
            '<%s>' % o for o in self.positional_options)
        self.parser = OptionParser('%%prog [options] %s' % positional_string,
                                   prog=func.__name__)
        for args, kwargs in self.optparse_options:
            self.parser.add_option(*args, **kwargs)

    def get_options(self, func):
        code = func.func_code
        return [ vn for vn in code.co_varnames[:code.co_argcount] ]

    @classmethod
    def run(klass, args):
        if args and args[0] in klass.commands:
            return klass.commands[args[0]](args[1:])
        else:
            print 'Unrecognized command'
            print 'Acceptable commands:'
            for name in klass.commands:
                print '  -', name

    def __call__(self, args):
        (opts,a) = self.parser.parse_args(args)
        if len(a) != len(self.positional_options):
            self.parser.error('Wrong number of arguments')
        o = {}
        for name in self.optparse_option_names:
            o[name] = getattr(opts, name, None)
        for name, value in zip(self.positional_options, a):
            o[name] = value
        return self.func(**o)      

Once I've built the decorator, the commands are fairly straightforward (as the add command shows). Here are the "short" commands:

@Command
def initialize():
    try:
        metadata.drop_all()
    except:
        pass
    metadata.create_all()
    print 'Service database initialized'

@Command
def list():
    q = Process.query()
    if q.count():
        for p in Process.query():
            print p
    else:
        print 'No processes'

@Command
def add(service, command, working_directory,
        stdin, stdout, stderr):
    p = Process(name=service,
                command_line=command,
                working_directory=working_directory,
                stdin=stdin,
                stdout=stdout,
                stderr=stderr)
    session.commit()

@Command
def remove(service):
    p = Process.query.get(service)
    session.delete(p)
    print 'Removed %s from the service list' % service
    session.commit()

@Command
def status(service):
    p = Process.query.get(service)
    print p      

Starting and stopping processes is a little more complex, but not too bad. First off, I needed a way to determine if a process was running. I didn't want to parse the ps -furick446 results, so I send a unix signal 0 to the PID (which doesn't do anything to the receiving process). If there's an exception, the process is either not running or not owned by me. So here's the is_running code:

def is_running(pid):
    try:
        os.kill(pid, 0)
        return True
    except Exception, ex:
        return False      

I also need a daemonizer function that will start, daemonize, and set the PID of a process object:

def daemonize(p):
    pid = os.fork()
    if pid: return # exit first parent
    os.chdir(p.working_directory)
    os.umask(0)
    os.setsid()
    pid = os.fork()
    if pid:
        Process.query.get(p.name).pid = pid
        session.commit()
        sys.exit(1) # exit second parent
    si = open(p.stdin, 'r')
    so = open(p.stdout, 'a+')
    se = open(p.stderr, 'a+', 0)
    os.dup2(si.fileno(), sys.stdin.fileno())
    os.dup2(so.fileno(), sys.stdout.fileno())
    os.dup2(se.fileno(), sys.stderr.fileno())
    args = shlex.split(p.command_line.encode('utf-8'))
    os.execvp(args[0], args)      

Once these are defined, I can start, stop, and restart processes fairly simply:

@Command
def start(service):
    print 'Starting %s ...' % service
    p = Process.query.get(service)
    if p.pid is not None:
        if is_running(p.pid):
            print '... %s already running with PID %s' % (
                service, p.pid)
            return
    p.pid = 0
    session.commit()
    daemonize(p)
    print '... started %s' % service

@Command
def stop(service):
    print 'Stopping %s ...' % service
    p = Process.query.get(service)
    if not p.pid or not is_running(p.pid):
        print '... service %s is already stopped' % service
        p.pid = None
        session.commit()
        return
    for retry in range(5):
        print '... sending SIGTERM to %s' % p.pid
        os.kill(p.pid, signal.SIGTERM)
        time.sleep(0.5)
        if not is_running(p.pid):
            break
    else:
        print '... sending SIGKILL to %s' % p.pid
        os.kill(p.pid, signal.SIGKILL)
        time.sleep(0.5)
        if is_running(p.pid):
            print '... process %s could not be killed' % p.pid
            return
    p.pid = None
    session.commit()
    print '... %s is stopped' % service

@Command
def restart(service):
    stop([service])
    start([service])        

Finally, I need to hook the script up and make sure it runs from the command line:

def main():
    Command.run(sys.argv[1:])

if __name__ == '__main__':
    main()        

And there you have it! A userspace daemonizer that lets you manage an arbitrary number of services. It is definitely overkill in many ways, but hopefully the sharing the process of building it will be as educational to you as it was to me.

Miruku - Migrations for SQLALchemy

One of the painful things about working with any database-oriented project in production is that you can't just drop the database and re-create every time you have a schema change. (Of course, you could do that, but your users might get a little miffed when their data disappears.) Rails and Django have for this reason had support for migrating from one schema version to another. Well, now SQLAlchemy has a automatic migrations tool by the name of Miruku. I haven't tried it, and it's an extremely early version (0.1a7), but it looks promising. Have a look here, and let me know what you think.