Announcing MetaPython - Macros for Python

As I mentioned in my last post, I have been considering writing some version of macros for Python and was looking for use cases. Well, having gotten the use cases I so desired from my wonderful commenters, I went ahead and put together an import hook and Google Code project that I'm calling MetaPython — all just in time for PyCon! (I have no talks, but I will be there, and would love to have a MetaPython Open Space if anyone's interested.)

So what's all the excitement about? MetaPython introduces some hooks to allow you to modify module code just before it is seen by the Python interpreter (at what I'm calling "import time"). The import-time syntax is pretty simple, and is (almost) all denoted by a question mark ? prefix (question marks are currently syntax errors in regular Python). Here is a trivial example that defines an import-time function (which as we will see can be used as a macro) that will conditionally remove a function call (and the evaluation of its associated arguments). Suppose the following text is saved in a file "cremove.mpy":

def cremove(debug, expr):
    debug = eval(str(debug))
    defcode empty_result:
        pass
    if debug:
        result = expr
    else:
        result = empty_result
    return result

The idea here is that cremove will be called with two metapython.Code values, debug and expr. cremove will convert debug to its Python code representation by calling str() and then evaluate the result. If debug is true, then expr will be returned. Otherwise a pass statement will be returned (defined using the MetaPython import-time construct defcode which defines a code template). To actually call cremove as a macro, we will need to define another MetaPython module, say "test_cremove.mpy":

import logging
logging.basicConfig(level=logging.DEBUG)
log = logging.getLogger(__name__)

?from cremove import cremove

def do_test():
    ?cremove(True, log.debug('This statement will be logged'))
    ?cremove(False, log.debug('This statement will not be logged'))

Here, we do an import-time import (the ?from... import business). This makes the module we're importing available at import-time (a regular import would be seen just as another line of Python code at import-time). To actually call cremove as a macro, we just need to prefix it with the "?" as shown.

Now, to actually test this, we'll need to install MetaPython and fire up an interpreter. MetaPython is available from the CheeseShop, so to get it just run easy_install MetaPython. Once it's installed, we can test our MetaPython code as follows:

>>> import metapython; metapython.install_import_hook()
>>> import test_cremove
>>> test_cremove.do_test()
DEBUG:test_cremove:This statement will be logged

Since macro expansion can get pretty complex and it's always tricky debugging code you've never seen, the fully-expanded module is available as the __expanded__ attribute of the MetaPython module:

>>> print test_cremove.__expanded__
import logging 
logging .basicConfig (level =logging .DEBUG )
log =logging .getLogger (__name__ )


def do_test ():
    log .debug ('This statement will be logged')
    pass  

There's a lot more to MetaPython, but hopefully this has whetted your appetite. There's a short tutorial available that shows how you can implement the collections.namedtuple class factory using a macro. It also shows how you can use Jinja2 syntax along with the defcode construct to dynamically produce Python code. Have fun, and let me know what you think!

Python Macros?

I've been thinking a bit about macros and what use they might be in Python. Basically, I was contemplating writing an import hook that would allow you to use code quoting and unquoting and stuff for your Python modules. My motive was just that Lisp people seem to rave about how awesome macros are all the time, so I figured they must be cool.

As I sat down to actually start figuring out what macro definitions and uses should look like in Python, I thought, hey, I'll just throw together a use case. But I haven't been able to come up with one (yet).

Most of the examples I found on the web focused on "hey, you can implement a 'while' loop with macros in Lisp!" or "hey, look at all the cool stuff the 'setf' macro can do!" So I started to wonder whether maybe Lisp people love macros because it allows them to extend Lisp's minimalist syntax with new constructs (like object-oriented programming with CLOS, while loops, etc.) Python, OTOH, has pretty rich syntax. It has a nice OOP system with syntactic support, while and for loops, generators, iterators, context managers, primitive coroutines, comprehensions, destructuring bind,.... -- What would I use macros for? (OK, depending on the syntax, I could add a "switch" statement, but that hardly seems worth the trouble.)

I should mention that I also saw some examples of people using macros for performance; you basically get rid of a function call and you can potentially make the inner loop of some critical function run really fast. But if that's all it buys me in Python-land (well, that and a switch statement), my motivation is pretty low. Because let's face it -- if your critical inner loop is written in pure Python, you can pretty easily throw it at Cython and get better performance than Python macros could ever provide.

So here's the question: does anyone out there have an idea of what macros would add to Python's power or expressiveness? Or maybe some Lisp, Meta OCAML, or Template Haskell hackers who can enlighten me as to what macros can add to a language with already rich syntax?

Update 2008-03-19

I have implemented MetaPython 0.1, a macro and code quoting system for Python, covered in the next blog post.