Thursday, March 12, 2009

Python Macros?

I've been thinking a bit about macros and what use they might be in Python. Basically, I was contemplating writing an import hook that would allow you to use code quoting and unquoting and stuff for your Python modules. My motive was just that Lisp people seem to rave about how awesome macros are all the time, so I figured they must be cool.

As I sat down to actually start figuring out what macro definitions and uses should look like in Python, I thought, hey, I'll just throw together a use case. But I haven't been able to come up with one (yet).

Most of the examples I found on the web focused on "hey, you can implement a 'while' loop with macros in Lisp!" or "hey, look at all the cool stuff the 'setf' macro can do!" So I started to wonder whether maybe Lisp people love macros because it allows them to extend Lisp's minimalist syntax with new constructs (like object-oriented programming with CLOS, while loops, etc.) Python, OTOH, has pretty rich syntax. It has a nice OOP system with syntactic support, while and for loops, generators, iterators, context managers, primitive coroutines, comprehensions, destructuring bind,.... -- What would I use macros for? (OK, depending on the syntax, I could add a "switch" statement, but that hardly seems worth the trouble.)

I should mention that I also saw some examples of people using macros for performance; you basically get rid of a function call and you can potentially make the inner loop of some critical function run really fast. But if that's all it buys me in Python-land (well, that and a switch statement), my motivation is pretty low. Because let's face it -- if your critical inner loop is written in pure Python, you can pretty easily throw it at Cython and get better performance than Python macros could ever provide.

So here's the question: does anyone out there have an idea of what macros would add to Python's power or expressiveness? Or maybe some Lisp, Meta OCAML, or Template Haskell hackers who can enlighten me as to what macros can add to a language with already rich syntax?

Update 2008-03-19


I have implemented MetaPython 0.1, a macro and code quoting system for Python, covered in the next blog post.

22 comments:

  1. I'm learning PLT Scheme right now and while the IDE is quite cool, I don't see much point to macros either.

    I think macros could create idiosyncratic constructs, which not pythonic.

    ReplyDelete
  2. Anonymous11:41 AM

    Macros could allow you to not use explicit self in every class method...

    ReplyDelete
  3. @anonymous: Not that I think it's a good idea, but you can actually do that without macros using metaclasses.

    ReplyDelete
  4. A coworker and I were just discussing this the other day...

    In C++ development at my previous employment, we used macros all the time, to reduce repeated code patterns. The most common example was in exception handling, for example, catching, logging, and then re-throwing an exception.

    As I was writing "except Exception, e: ..." for yet another time the other day, I realized that with macros I could have avoided this repetition. Avoiding repetition means I only have to write it once, and thus, less potential for bugs!

    ReplyDelete
  5. @chphilli: I kind of see what you're saying, but couldn't you achieve the same thing with either a decorator (to guard a function) or a context manager (to guard an arbitrary block of code)?

    with logging_exceptions:
        some
        potentially
        exception
        raising
        stuff

    ReplyDelete
  6. Yeah, I suppose you could -- I like the idea of a context manager approach -- in fact, I'll probably implement it that way sometime next week now that you point it out.

    I think the original point probably still stands though -- any where you have a set of boiler-plate code, it may be nice to have macro support.

    That said, as you've pointed out, it's nowhere near as important as in less flexible languages like C++!

    ReplyDelete
  7. Anonymous3:17 PM

    I guess Lisp programmers when creating templating or configuration languages want to keep programming in Lisp plus a few additions, whereas Python programmers either need something completely different so that a few macros would not help them much anyway or if they want Python they want to stay as close as possible to its syntax.

    ReplyDelete
  8. Macros allow your code-definition code to look more like regular code.

    The type() example from last night's PyATL meetup is a good example:

    class Test{BROWSER}(BrowserTestCase):
     browser = {BROWSER}

    looks a lot more like a normal Python class definition than:

    type("Test%s" %(CLASSNAME, BROWSER), (BrowserTestCase,), dict(browser=BROWSER))

    ReplyDelete
  9. @Zellyn: I see what you're saying, and I agree that the type(...) approach can be off-putting, but I can also just write the following:

    class Test(BrowserTestCase):
        browser = browser
    Test.__name__ = 'Test%s' % browser

    In Python 2.6, I could even write a class decorator to clean it up:

    @setname('Test%s' % browser)
    class Test(BrowserTestCase):
        browser = browser

    where I have

    def setname(name):
        def inner(o):
            o.__name__ = name
            return o
        return inner

    Another thing I didn't like about the type(...) approach is that it doesn't account for metaclasses, but that's a minor pet peeve.

    Actually, since you mentioned this, I can now think of a good use case for macros -- it would clean up the collections.namedtuple class generator quite a bit. Right now, it puts together a string and eval()s it. Kind of a poor man's macro anyway.

    ReplyDelete
  10. Anonymous10:00 PM

    Rick, it was good seeing you last night. I think Python and Lisp are pretty close in terms of expressiveness. They have different philosophies though, and it attributes to why/how they are different. I much prefer Python's syntax to Lisp - code is generally much more pleasant to read, which is a big win. On the other hand, I like the simplicity of Lisp, in that there's just not that much to it. Macro's are so powerful you can do almost anything with it. On the other hand in Python, you have a large set of tools at your disposal(metaclasses, decorators, generators, the *with* clause, etc.). There's more you need to learn and more to chose from. Also, since they are language features, you can't extend them yourself - we have to design them by committee. Python wasn't always this expressive - It took many years to get to this point. Lispers were already enjoying their Macros almost at the beginning. Back then, there was nothing close to it, so it's no wonder they are so enthusiastic about them. Things are a bit different now - other languages have caught up.

    ReplyDelete
  11. Macros are great for conditionally ignoring a section of code. The one particular use I have in mind is for debug logging. Our massive code project is littered with function calls to do debugging, protected by `if trace_level` conditionals to prevent evaluation of the function call and its (sometimes obscenely large) argument payload. Two of the most expensive things one can do in Python are create objects and call functions, and logging generally does both.

    ReplyDelete
  12. @Scott: I suppose I can see using macros to differentiate between debug and optimized code, although it looks like you have a passable workaround with simple Python conditionals. It seems that a macro facility would probably make your "massive code project" code look cleaner (i.e. without the conditionals).

    Between that example and collections.namedtuple, I'm just about convinced that macros could add something to Python. After reading the comments here, however, I am also convinced that Python needs them a lot less than a minimalist language like Lisp.

    ReplyDelete
  13. The point of macros is that you can extend the language without cooperation from the core developers. Consider the new Python "with" statement. If Python had macros, not only could this have been implemented outside the core (rather than waiting for a committee to debate it for months), it could easily be backported to older versions of Python.

    Python has such a rich array of features precisely because it must. Since it's impossible to extend the language in any significant sense, it must implement every useful feature it can. Some of us think of this as "bloat".

    ReplyDelete
  14. ...and now I see MetaPython... nice work! Don't know if it's enough to keep me in Python-land for the long-term, but definitely cool.

    ReplyDelete
  15. I'm just starting Python, and I already found a good use case for Macros.

    I want to run a test which is expected to raise an exception and write it like this:

    test-exn("exception content", func(args));

    Namely, I want this to expand to a try/catch statement without having to write the try/catch, and without having to write a class to encapsulate unit tests. These are for small scripts, but they're large enough that I want to have a small bit of regression testing. Perhaps I'm writing a miniature interpreter...

    Lisp macros can (and do) do this.

    ReplyDelete
  16. @jeremy:

    The way this is typically done in python is to write an assertion function (this is how the unittest module does it):

    assert_raises(ExceptionType, func, *args, **kwargs)

    where the assert_raises function is something like:

    def assert_raises(t, func, *args, **kwargs):
      try:
        func(*args, **kwargs)
        assert False, '%s not raised' % t
      except t:
        return
      except:
        raise

    But I can definitely see how macros could be useful, as well.

    ReplyDelete
  17. Anonymous5:36 PM

    http://www.cs.brown.edu/~sk/Publications/Talks/SwineBeforePerl/

    Watch the presentation and the slides about implementing a Finite Automata using Scheme Macros.

    ReplyDelete
  18. Anonymous5:15 PM

    decent for macro'ed with a while

    ReplyDelete
  19. I want to replace for with a parallel_for that does something else with the block underneath the for. In order to do that with "with" I need to bracket the "for" with "withs"

    with parallel_sync():
    for i in range(0,4):
    with parallel_split():
    ...closure that will have each
    instance of the "for" run in
    ...parallel

    This is the only way I can know,
    1)each time I enter and leave an individual loop of the for block
    2)when I leave the entire for block

    ReplyDelete
  20. Anonymous9:48 PM

    I found this post because I wanted to implement functional conditions so that I could use them in a lambda statement, because

    ifelse(cond, t, f)

    reads easier than

    cond and t or f

    People dont usually expect they will be used like that.

    The problem is that defining them as functions cause both t and f to be evaluated before the condition. All the arguments are executed, and the return values passed into the conditional function, where the cond is then used in an if, and the appropriate return is returned.

    Because of this, I realised that I wanted to use a macro to declare the function, using the and or trick.

    ReplyDelete
  21. I've kind of abandoned this project (it's still available at http://code.google.com/p/metapython/ if anyone's interested in looking at/picking it up). For something like ifelse, recent versions of python gave you the ternary expression:

    t if cond else f

    which is marginally more readable (and more semantically correct) than the old

    cond and t or f

    Anyway, that's my $.02 worth.

    ReplyDelete
  22. Anonymous12:20 AM

    Macros are for extending your compiler. I can implement a syntax transformation that can drastically change the semantics of a certain piece of code, without having to actually go in and hack the compiler to support the new feature in question.

    For example, I hacked together a simple tail-recursion macro today just for the hell of it.

    https://gist.github.com/3967055

    Obviously pointless and just for fun, but I know of few languages that could support something like this without a new version number.

    ReplyDelete