Friday, July 18, 2008

RESTfulness in TurboGears

Mark Ramm and I were talking about how to differentiate GET, POST, PUT, and DELETE in TurboGears 2 and we came up with a syntax that's pretty cool. That's not the reason for this post, though. This morning, I noticed that our syntax is completely compatible with TurboGears 1.x -- so here's how you do it.

What we wanted was to expose a RESTful method like this:

class Root(controllers.RootController):

class index(RestMethod):
def get(self, **kw):
return dict(method='GET', args=kw)
def post(self, **kw):
return dict(method='POST', args=kw)
def put(self, **kw):
return dict(method='PUT', args=kw)
# NOT exposed, for some reason
def delete(self, **kw):
return dict(method='DELETE', args=kw)

The TurboGears 1 implementation relies on the way that CherryPy determines whether a given property is a valid URL controller. It basically looks at the property and checks to make sure that:

  • it's callable, and

  • it has a property exposed which is true (or

The "a-ha!" moment came when realizing that:

  • Classes are callable in Python (calling a class == instantiating an object)

  • Classes can have exposed attributes

So with appropriate trickery behind the scenes, the above syntax should work as-is. So here's the "appropriate trickery behind the scenes":

class RestMeta(type):
def __new__(meta,name,bases,dct):
cls = type.__new__(meta, name, bases, dct)
allowed_methods = cls.allowed_methods = {}
for name, value in dct.items():
if callable(value) and getattr(value, 'exposed', False):
allowed_methods[name] = value
return cls

The first thing I wanted to do was create a metaclass to use for RestMethod so that I could save the allowed HTTP methods. Nothing too complicated here.

import cherrypy as cp
class ExposedDescriptor(object):
def __get__(self, obj, cls=None):
if cls is None: cls = obj
allowed_methods = cls.allowed_methods
cp_methodname = cp.request.method
methodname = cp_methodname.lower()
if methodname not in allowed_methods:
raise cp.HTTPError(405, '%s not allowed on %s' % (
cp_methodname, cp.request.browser_url))
return True

This next thing is tricky. If you don't understand what a "descriptor" is, I suggest the very nice description here. The basic thing I get here is the ability to intercept a reference to a class attribute the same way the property() builtin intercepts references to object attributes.

The idea here is to use this descriptor as the exposed attribute on the RestMethod class. When CherryPy tries to figure out if the method is exposed, it calls ExposedDescriptor.__get__ and uses the result as the value of exposed. If the HTTP method in question is not exposed, then the code raises a nice HTTP 405 error, which is the correct response to sending, say, a POST to a method expecting only GETs.

The final part of the solution, the actual RestMethod, is actually pretty simple:

class RestMethod(object):
__metaclass__ = RestMeta

exposed = ExposedDescriptor()

def __init__(self, *l, **kw):
methodname = cp.request.method.lower()
method = self.allowed_methods[methodname]
self.result = method(self, *l, **kw)

def __iter__(self):
return iter(self.result)

The sequence of things is now this:

  • CherryPy traverses along to the root.index class and looks up

  • is intercepted by the descriptor which checks the CherryPy request method to see if it's valid for this controller, and if it is, returns True

  • CherryPy says, "Great! root.index is exposed. So now I'll call root.index(...)." This calls index's constructor, which in turn calls the appropriate method and saves the result in self.result

  • CherryPy says "Cool! root.index returned me an iterable object. I'll iterate over it to get the text to send back to the browser." This calls root.index(...).__iter__, which is just delegated to the result that the real controller gave.

At then end, we get a fully REST compliant controller with a nice (I think) syntax.

Update 2008-07-21: After some more thinking, I realized that the metaclass isn't necessary. The descriptor is more important, but also not strictly necessary. My original design did everything in the constructor of RestMethod, but this runs after all of TurboGears' validation and identity checking happens, so it's pretty inefficient. If you want the implementation with the descriptor but without the metaclass, you can do this:

import cherrypy as cp
class ExposedDescriptor(object):
def __get__(self, obj, cls=None):
if cls is None: cls = obj
cp_methodname = cp.request.method
methodname = cp_methodname.lower()
method = getattr(cls, methodname, None)
if callable(method) and getattr(method, 'exposed', False):
return True
raise cp.HTTPError(405, '%s not allowed on %s' % (
cp_methodname, cp.request.browser_url))

The benefit to using a metaclass is that the check for the existence, "callability", and exposed-ness of the method happens up front rather than on each request. Also, I don't like the fact that the metaclass pollutes the class namespace with the "allowed_methods" attribute. There's probably a way to clean that up and put the descriptor in a closure, but I haven't had time to look at it. Maybe that will be a future post....


  1. When this will be in the tg2 repository?

  2. Anonymous4:04 PM

    Does the TurboGears 2 version also need a metaclass? Grok's REST support looks quite similar, except we don't need a metaclass nor expose decorators.

    Do you have support for exposing the same model through REST and non-REST both? Or multiple REST protocols for the same model?

  3. The tg2 version shouldn't need any metaclass magic, as we have control of the object lookup mechanism and can inspect the RestMethod subclass. As for when it will be in the tg2 repository, Mark's adding a ticket now.... :-)

    As for comparisons to Grok, I can't really comment in detail because I'm not too familiar with Grok. In TurboGears, we don't actually expose the model objects; we expose a tree of controller objects which may (or may not) interact with the model. So it would be simple to create two controllers that affected the same model; you just create two controllers.

  4. That's a pretty cool recipe!

    I would love to have that somewhere on the TurboGears doc wiki. Would you mind putting it up there? The best thing, of course, would be a small, functional example project.

    If you don't have time to do this, and you don't mind, I can put up a page for this recipe, when I have time.

    Chris (TurboGears sysadmin)

  5. @Chris:

    I certainly don't mind you putting it up on the wiki. How about whenever one of us has time, it goes up?

  6. CP 3 lets you register a custom dispatch function, so metaclasses shouldn't be necessary in TG 1.5, either.

    If Mark commits some tests with his ticket or you or Chris creates a 1.x recipe page, I'll try to port it to 1.5.

  7. @ken:

    I realized that the metaclass magic isn't strictly necessary here, either, and have added a note to the post to indicate that.

  8. Nice. Glad to see turbogears progressing towards this sort of feature set.

    I'm just curious why you didn't use or mention the built in cherrypy MethodDispatcher which, AFAICT, does the basic same thing?

  9. @Lakin:

    Full disclosure: The main reason I didn't use MethodDispatcher is that I didn't know about it. ;-)

    Also, the TurboGears 1.0.x release series runs under CherryPy 2.x, not CherryPy 3, which is where (I believe) the MethodDispatcher was introduced. This recipe should work for the older CherryPy.

  10. Anonymous7:28 PM

    Rick: grok uses a model/view separation, where our view is much like your controller. except that Grok models typically don't get looked up by the views as the view is already acting for a context model object (due to Grok's traversal through content, not code).

    With TG's controller mechanism I imagine the typical approach would be to just build two entirely disparate URL spaces if you want to expose the same models through two REST protocols, or a REST protocol and a non-REST protocol.

    In Grok you can accomplish that too, but besides this, essentially the same resources (in URL space) can be approached using multiple UI skins or REST protocols as well.

    Grok's REST support has been released for quite a while now. :)

  11. I've tried to use this pattern in a TurboGears 1 app, and I've written a fairly substantial test suite. Where can I post it for it to be useful?

  12. @Peter:

    Thanks for the contribution! Luckily, Mark Ramm happened to be sitting in the same room as me when I saw your comment, so I asked him. His recommendation is to file a ticket at with the tests as an attachment, and it will probably be added to TG 1.1.

  13. @Rick

    I've added the ticket. It's #1996