Thursday, March 19, 2009

Announcing MetaPython - Macros for Python

As I mentioned in my last post, I have been considering writing some version of macros for Python and was looking for use cases. Well, having gotten the use cases I so desired from my wonderful commenters, I went ahead and put together an import hook and Google Code project that I'm calling MetaPython — all just in time for PyCon! (I have no talks, but I will be there, and would love to have a MetaPython Open Space if anyone's interested.)

So what's all the excitement about? MetaPython introduces some hooks to allow you to modify module code just before it is seen by the Python interpreter (at what I'm calling "import time"). The import-time syntax is pretty simple, and is (almost) all denoted by a question mark ? prefix (question marks are currently syntax errors in regular Python). Here is a trivial example that defines an import-time function (which as we will see can be used as a macro) that will conditionally remove a function call (and the evaluation of its associated arguments). Suppose the following text is saved in a file "cremove.mpy":

def cremove(debug, expr):
debug = eval(str(debug))
defcode empty_result:
if debug:
result = expr
result = empty_result
return result

The idea here is that cremove will be called with two metapython.Code values, debug and expr. cremove will convert debug to its Python code representation by calling str() and then evaluate the result. If debug is true, then expr will be returned. Otherwise a pass statement will be returned (defined using the MetaPython import-time construct defcode which defines a code template). To actually call cremove as a macro, we will need to define another MetaPython module, say "test_cremove.mpy":

import logging
log = logging.getLogger(__name__)

?from cremove import cremove

def do_test():
?cremove(True, log.debug('This statement will be logged'))
?cremove(False, log.debug('This statement will not be logged'))

Here, we do an import-time import (the ?from... import business). This makes the module we're importing available at import-time (a regular import would be seen just as another line of Python code at import-time). To actually call cremove as a macro, we just need to prefix it with the "?" as shown.

Now, to actually test this, we'll need to install MetaPython and fire up an interpreter. MetaPython is available from the CheeseShop, so to get it just run easy_install MetaPython. Once it's installed, we can test our MetaPython code as follows:

>>> import metapython; metapython.install_import_hook()
>>> import test_cremove
>>> test_cremove.do_test()
DEBUG:test_cremove:This statement will be logged

Since macro expansion can get pretty complex and it's always tricky debugging code you've never seen, the fully-expanded module is available as the __expanded__ attribute of the MetaPython module:

>>> print test_cremove.__expanded__
import logging
logging .basicConfig (level =logging .DEBUG )
log =logging .getLogger (__name__ )

def do_test ():
log .debug ('This statement will be logged')

There's a lot more to MetaPython, but hopefully this has whetted your appetite. There's a short tutorial available that shows how you can implement the collections.namedtuple class factory using a macro. It also shows how you can use Jinja2 syntax along with the defcode construct to dynamically produce Python code. Have fun, and let me know what you think!


  1. I see you are having fun in your vacations!

  2. Anonymous12:51 PM

    Please provide the full posts feed to planet python like just about everyone else.

  3. @anonymous: Sorry about that; I think I fixed the feed in Blogger. Not sure how to go about updating anything on planet Python, however.

  4. Anonymous2:13 PM

    Peter Norvig's lessons on Lisp programming might be something to consider:

    6. Use macros (if really necessary). [p. 66]

    52. A word to the wise: don't get carried away with macros [p. 855].

    In a recent thread, several folks who had experience with "large" programming teams raised an interesting point along the lines of:

    Where Java succeeded that so many fail to acknowledge is in the area of team development. It's a language that says you should have the power of a given feature but only if that feature doesn't create problems for other people trying to read and maintain your code. Note at this point that I believe that it has had some significant failures in this regard.

    If we want to see Python gain more than a niche foothold, it might be a good idea to not add a feature that even seasoned Lisp hackers aren't always comfortable with.

  5. @sk: I definitely see your point there. That is one reason I wanted to make sure that macros and functions were different at the call site. If you have someone who's not familiar with MetaPython, the first thing they will notice is the .mpy extension, then the ?foo(...) syntax, at which point they'll realize, "Hey, something strange is going on here that I should investigate."

    With Lisp, a macro call is identical to a function call, so all bets are off. Beginners don't know which forms are functions and which are macros, making the learning curve quite steep.

    A big motivation for me to do this (MetaPython) is the current implementation of collections.namedtuple in the standard library. It uses string pasting to get some Python code to be exec'd in the calling module's environment, something I'd wager is even less easy to grok than macros. (That's why the tutorial shows how you can implement collections.namedtuple with macros.)

    Oh, and I'm not proposing this as a PEP or anything -- I'd like to see the base Python syntax stay as it is. But I think it could be useful in some cases.

    Also, FWIW, I really don't think you can consider Python to be a "niche" language any longer. Sure, it's nowhere near as widespread as the "enterprise" languages like C++, C#, Java, etc., but it seems to have a larger footprint than Lisp, Smalltalk, Haskell, O'Caml, etc. (But feel free to correct me if I'm wrong!)

  6. Anonymous2:55 PM

    I did a bit of source code scanning from several decent-sized code bases for a presentation on Class Decorators (PDF, might have to right click and save) and came across few usage examples of function decorators, other than the usual @classmethod. Unfortunately, the fact remains that even decorators or functools methods are a bit of a stumbling block for the majority of Python developers, esp. those without any background in Functional Programming. AFAIK, there aren't many Python based projects that employ teams of, say, 30+ developers (a size that's not unusual at all in enterprise projects). Unfortunately, Python is still used very rarely for projects of that scale, and like non-academic languages such as Lisp and Smalltalk, it still suffers from the perception that, the necessarily niche, project in it has to be driven by a wizard-level person assisted by a small team.

    Python has to break the perception barrier which posits that "dynamic languages such as Python, Ruby and Groovy are better for small and medium applications development than static languages such as C++ and Java" for it to ever become a language with a double-digit following in the marketplace. For this to happen, exotic feature implementation might have to be put on the back burner compared with more mundane but vital tasks such as maintenance on the Standard Library that absolutely needs a serious 'D check' as they say in the aviation industry.

    At any rate, I and others (based on reduced attendance at Pycon this year which I'm told might lose hundreds of thousands for PSF) could certainly use a larger demand for Python programmers, even with the downturn.

  7. @sk: I really can't comment much on projects of 30+ developers. I can say that in my experience, Python's productivity gains over static, imperative languages yield a 5-10x efficiency increase (for me personally). So perhaps the reason for the non-use of Python in large (30+) development teams is that it would take only 3-5 Python programmers to accomplish what 30+ Java programmers could (*ducks*) ;-).

    I did read the page you linked to re: dynamic languages being better for small/medium projects, and it does say that, but it does not say anything about larger projects (probably because the case is less clear). And the presentation referenced on your linked page actually didn't say anything about problem scale. In fact, its summary included the following choice quote:

    Python works well for all applications, especially platform independent GUIs.

    As to the standard library, it seems that the core language team agrees with you, and the library has undergone significant reorganization for Python 3.0.

    As for PyCon attendance, it is indeed reduced this year, probably due to the economy. But we need to keep in mind that there are already more paid registrations this year (836 as of 3/19/09) than the number of attendees in 2008 (586).

    As for diverting effort toward "exotic feature implementation," I am not and never have been a core Python language or standard library developer. Maybe someday I will get into it. For now, this was something I put together on my vacation (as Laurent mentioned above) because I thought it would be useful in some cases.

  8. Anonymous4:24 PM

    Those are the conclusions that Russel Winder reaches in his presentation (and given his consulting business are perhaps a tad bit to be expected ;) but the perception of Python remaining stuck in a niche—or "ghetto" as he provocatively put it elsewhere—is still real and is indicated in his very framing of the issue. Python use has been stable, but frankly given the free-fall in Perl "mindshare", we could have done a lot better.

    It's true that Python productivity is higher, but based on several experiences it's still an uphill struggle getting most large organizations—those with 7 figure project budgets and larger development staffs—to commit to Python for their core products, even if they realize the savings due to higher productivity of individuals. That was one of the USPs (Unique Selling Point) of Java as well, and it had a strong presence in the majority of development shops within 5 years of its introduction. We have to ask ourselves what are the reasons for the lack of penetration by Python. Unfortunately, things like not having a smooth and up-to-date library hurt acceptance and I'm afraid PEP 3108 doesn't go far enough in addressing the changes needed—which in any case will not take effect for a minimum of 3 years given the 3.0 timeframe.

    Chicago Pycon attendance numbers are significantly higher than those for the DC and Dallas conferences and last year's numbers led the organizers to make a perhaps too optimistic a projection for this year, hence the financial risk. Let's hope it's mostly due to the economy.

    Ideas one gets during a vacation can turn out to be revolutionary!

  9. @sk: I didn't catch the presentation, but I did see the description of Winder's talk at the link, and it looks like he's asking the question of whether it's a ghetto, not asserting that it is.

    As far as corporate support Python goes, I can't tell you exactly why organizations do what they do, but I can guess that Java's success compared to Python's has a lot more to do with the backing of Sun than the actual technical merits of the language.

    Another aspect of Java is that it's a great "leveler" of programmers -- there seems to be (IMO) a smaller difference between the quality/productivity of the best and worst Java programmers than there is between the best and worst [C++/Lisp/Python/Lua/Ruby/Scheme/Smalltalk/anything else but maybe C# and VB] programmers. And let's face it, if your organization needs developers to crank out mundane, complex but ultimately uninteresting enterprise code, you're probably not going to attract the best developers. So you're probably more interested in how the mediocre ones do while using your language. Java productivity is more predictable (albeit significantly lower) than Python productivity, and predictability is sometimes more important than raw productivity.

    Anyway, good comments! Just curious -- did you have a chance to look at MetaPython on technical (rather than philosophical/advocacy) grounds? What was your opinion of it there?

  10. Anonymous6:07 PM

    Perhaps I should have framed Winder's comment as a rhetorical question instead of a statement, but such a question is sometimes more revealing than a dry point of query (e.g. if anybody asks "Is .NET dying?", they had better have a solid basis for that question, otherwise they'd be laughed out of the room, or at least chatroom).

    You might be surprised at the level of development talent within some large organizations as well. After all, someone has to actually have ideas that earn the company money. In fact, someone I met not long ago who championed Python in his organization mentioned the difficulty in selling Python to upper management because of the "boutique" development type of perception issues it is saddled with, which also touch affects things like recruitment.

    I looked at the code you put up and other than specialized needs of framework developers, it didn't seem like something that would serve the "end-user" programmer's needs much, esp. given the rich constructs already available. But, maybe there are use cases that could serve a wider audience as well.

    The comment count on Bruce Eckel's blog on the legacy of Java and C++ is already well over a hundred, so I won't go there. ;)

  11. @sk: I agree that MetaPython is more for a framework/library developer, at least from a macro-writing perspective. "End-users" might use a macro defined in a library, but probably wouldn't be writing many of their own. (Of course, that is the case with Lisp, as well.) Thanks again for the comments.

  12. @sk, @myself:

    Upon re-reading my comment re: enterprise developers, I realize that I painted a lot of very talented developers with a very broad brush. Some of the best developers I've had the privilege to work with have been enterprise software developers. (Interestingly enough, the developers I'm thinking of were actually working in a mixed-language environment, programming in VB, C#, Python, and Smalltalk.)

  13. FYI, very interesting post and comments.