New Year's Resolutions

Categories: resolution, goals, fpga, lisp, turbogears, python, programming

Well, in the spirit of the season, I will give my top 10 technical-related New Years resolutions (or goals, or whatever name you prefer).

10. Sign at least 3 customers up for ConsulTracker.com - This should (hopefully) provide some low-effort recurring income, as well as get my web coding skills up to snuff. We'll say one customer by February, another by May, and a third by August.

9. Attend most (if not all) of the Python meet-ups in Atlanta - I really need to get more involved in the community. When I wrote my little TurboGears/Django/Subway comparison and got a comment from Kevin Dangoor (the main force behind TurboGears), it just about made my day. I always kind of looked at the big FOSS programmers as these uber-coder celebrities. Nice to know they're not inaccessible. And there might even be some contributions I could make.

8. Develop at least 2 new applications - Well, could be one new application and major feature enhancements for ConsulTracker. I'm thinking of a customizable workflow management system. Kind of like a bug tracker, but more generalized. It would let you take orders, route them to the warehouse, generate invoices, take payments, etc. It will be rule-based, have email notifications, and be scriptable in Python (for business rules). As to the major ConsulTracker feature enhancements, I'm looking at adding an expense reporting feature as well a traditional timeclock feature for hourly employees (not consultants). The timeclock would need a lot of custom logic for overtime, etc. I'd like to make it customizable to the individual business, but there's some thinking to do there.


7. Do something interesting in LISP - I'm really intrigued by the language, I use EMACS, I've copied and extended a LISP HTML templating library, but I've never really done anything real. I kind of look at languages in four productivity categories (roughly 3-10x improvement per class): C/C++ ==> Java/C# ==> Python/Ruby/Perl ==> LISP/Scheme. I can vouch for at least some of the languages in the first three categories, but I'd like to see what LISP can really buy me in productivity.

6. Publish GRACE technical report - This one really should have been done a few months ago when it was written, but look for it in the first few weeks of 2006.

5. Publish at least three papers as lead author - Not including the technical report, I really need to get into some peer-reviewed journals/conferences. I've submitted to DAC, but who knows if it'll make it. And that's three acceptances, not necessarily publications (for conferences in 2007, etc.). This will likely take the whole year (to get all three).

4. Extend the GRACE backend to handle all the little things I've been considering - like shared functions, shared functional units, bitwise cell characterization and full static timing analysis. This should get me from "competitive" with hand designs to "better most of the time." GRACE needs to be able to traverse the area/time design space in the micro (at the level of functional units), not just the macro (at the level of loops).

3. Extend the GRACE frontend to handle all the little things I've been considering- like (basic) object-oriented programming, better data structures, macros, array dependence analysis, and a real C/C++ front-end. Since the value proposition of GRACE is improving hardware designer productivity, the language level needs to be raised. For hardware, I see the productivity classes as VHDL/Verilog ==> Handel C ==> C/C++/SystemC/etc ==> HumIR (now) ==> HumIR (4Q2006). The goal is to improve HumIR enough to merit an increase in productivity class.

2. Do something interesting with GRACE - My research group is currently looking at using GRACE to synthesize (parts of) an H.264 encoder or decoder. That might satisfy this requirement, or maybe I need something more extensive. To get mindshare (and eventually marketshare), GRACE will need the archetypical "flashy demo." Small amount of HumIR code (less than 1000 lines, I'd say), FPGA or structured ASIC platform, pretty pictures.

1. Propose my thesis - Somewhat dependent upon the papers above, I need to at least have my thesis proposed by next December. That means doing a whole lot of other things as well, like re-enrolling in school, etc. It also means a lot of reading of papers. I hope to have a draft by July, final version by October, accepted by December. (Arbitrarily chosen dates, but at least something to shoot for.)

Consultant Tracker

Well, the consultant time-tracking application is just about feature-complete, and I'm about to start accepting beta customers! It's got project time management, integrated security, timesheet
entry and approval, accounts payable and receivable reporting, and the ability to delegate one userid to another user (without sharing passwords).

There are some obvious stylistic changes I'll be making before going "live" (post-beta), including using CSS, AJAX, etc. The good news about this version is that it works without AJAX, and it's fairly fast anyway. There's a test domain set up currently for anyone who wants to try it. Userid/password is test/test, so give it a try! Any feedback (positive or negative) is hereby actively solicited.
In addition, a big thank you goes out to Kevin Dangoor for creating TurboGears. It's helped a newbie to web programming like me get up and running very quickly. Thanks!

Hierarchical CherryPy

Forging ahead on my time tracking web application, I ran into a little problem with CherryPy/TurboGears. Specifically, it has to do with creating hierarchical URL trees that look like "http://.../customers/<customer_id>/". What I'd like to be able to do is something like this:

class Root(controllers.Root):
   def __init__(self):
       self.customers=Customers()

class Customers(controllers.Root):
   def lookup(self, custid):
       'Some lookup function which will continue traversing the URL'
       self.customer=Customer(custid)

class Customer(controllers.Root):
   @turbogears.expose(html="ct.templates.customer")
   def index(self, arg):
       ...

For now, I've overridden the __getattr__ method of the "Customers" class to do this, but it looks like TurboGears looks up the root.customers.cust_id about four times per request, so it seems a little inefficient. CherryPy's default method won't work either, because once you hit default, there's no more traversing. I think Quixote has something that handles this, but it's been a while since I looked at it. Any suggestions?

Categories: python, turbogears, web, cherrypy

Stupid Metaclass and Template Tricks

I've been playing around with various web templating solutions and really like the approach taken with Twisted/Nevow/Stan. Basically, you can create HTML by doing something like this: (taken from this example)

T.html[
       T.body[
            T.h1['ChildPage'],
            T.p['My name is: ', T.span(id="name")[render_name]],
            T.p[
                'I have child too: ',
                T.a(id="child", href=url.here.child('childOfChild'))['my child']
                ],
            ]
        ]

I like the programmatic construction and indentation features. The only problem is that I use emacs, and emacs doesn't like to indent nested Python lists like that. Some Quixote developers were also looking at extending their templating solution to include STAN-like blocks in this thread. I like the general approach, and I certainly like the resulting syntax, but I'm not crazy about using the compiler module and import hooks for web templates.

I've also been looking at using metaclasses for a while. I find them intriguing, and feel that my "Python cred" would be enhanced if I could actually come up with a decent use for them. Ian Bicking has written a few posts recently concerning them, and so I thought I'd try my hand at implementing something like what the Quixote developers are doing, but without using import hooks. (The only black magic here is the use of sys._getframe() and metaclasses.) I'm sure there are plenty of bugs, etc. in the code, but here it is for anyone who'd like to take it and run (or comment on how it could be improved). The actual template is at the bottom. Enjoy!

#!/usr/bin/env python2.4
import sys
from string import Template
from xml.sax import saxutils

def findParentFrame(start_depth = 0):
  f = sys._getframe(start_depth+1)
  while f.f_back  and 'meta' in f.f_locals  and issubclass(f.f_locals['meta'], AddToParentContent):
      f = f.f_back
  return f

class AddToParentContent(type):
  def __new__(meta, name, bases, dct):
      klass = type.__new__(meta, name, bases, dct)
      meta.addParentContent(klass)
      return klass
  @classmethod
  def addParentContent(meta, obj):
      f = findParentFrame()
      f.f_locals.setdefault('_content', []).append(obj)

class TagMeta(AddToParentContent):
  def __new__(meta, name, bases, dct):
      dct.setdefault('_content', [])
      dct.setdefault('_attrs', {})
      klass = AddToParentContent.__new__(meta, name, bases, dct)
      if klass.__doc__:
          klass._content.insert(0, _text(klass.__doc__))
      return klass
  def __init__(klass, name, bases, dct):
      AddToParentContent.__init__(klass, name, bases, dct)
      for k,v in dct.items():
          if k.startswith('_'): continue
          if isinstance(v, type): continue
          klass._attrs[k] = v

class Tag(object):
  __metaclass__=TagMeta
  def __init__(self, *l, **kw):
      self._l, self._kw = l, kw
  def __str__(self):
      attrs = ' '.join(['%s="%s"' % t for t in self._attrs.items()])
      l = [ '<%s %s>' % (self.__class__.__name__, attrs) ]
      content = []
      for obj in self._content:
          if callable(obj): obj = obj(*self._l, **self._kw)
          s = str(obj)
          s = s.replace('\n', '\n    ')
          content.append('    ' + s)
      l.extend(content)
      l.append('</%s>' % self.__class__.__name__)
      l = [ s for s in l if s ]
      return '\n'.join(l)

def _text(s, escape=True, substitute=True):
  tpl = Template(s)
  if escape and substitute:
      def inst(*l, **kw):
          return saxutils.escape(tpl.safe_substitute(*l, **kw))
  elif escape:
      def inst(*l, **kw): return saxutils.escape(s)
  elif substitute:
      def inst(*l, **kw): return tpl.safe_substitute(*l, **kw)
  else:
      def inst(*l, **kw): return s
  return inst

def text(s, escape=True, substitute=True):
  inst = _text(s, escape, substitute)
  f = findParentFrame(1)
  f.f_locals.setdefault('_content', []).append(inst)

def eval_(expr, escape=True, substitute=True):
  def inst(*l, **kw):
      inner = _text(str(eval(expr, kw, kw)), escape, substitute)
      return inner(*l, **kw)
  f = findParentFrame(1)
  f.f_locals.setdefault('_content', []).append(inst)

def foreach(seq):
  class _:
      __metaclass__=AddToParentContent
      def __init__(self, *l, **kw):
          self._l, self._kw = l, kw
      def __str__(self):
          iter_name = self.__class__.__name__
          l = []
          kw = self._kw.copy()
          for i in seq:
              kw[iter_name] = i
              for obj in self._content:
                  if callable(obj): obj = obj(*self._l, **kw)
                  l.append(str(obj))
          l = [ s for s in l if s ]
          return '\n'.join(l)
  return _

def if_(cond):
  class _:
      __metaclass__=AddToParentContent
      def __init__(self, *l, **kw):
          self._l, self._kw = l, kw
      def __str__(self):
          l = []
          if eval(cond, self._kw):
              for obj in self._content:
                  if callable(obj): obj = obj(*self._l, **self._kw)
                  l.append(str(obj))
          l = [ s for s in l if s ]
          return '\n'.join(l)
  return _

class html(Tag):
  class head(Tag):
      class title(Tag): text('$title')
  class body(Tag):
      class h1(Tag): text('$title')
      class a(Tag): "CNN"; href="http://www.cnn.com"
      class table(Tag):
          border=1
          class i(foreach(range(3))):
              class tr(Tag):
                  class j(foreach(range(3))):
                      class cond(if_('i <> j')):
                          class td(Tag): '$i > $j'
                      class cond(if_('i == j')):
                          class td(Tag): '$i = $j'
    
h = html(title="My Page")
print str(h)

Categories: python, metaclass, template, web, programming

Del.icio.us Account

Well, I just got an account on del.icio.us (finally!), so you can see what I'm reading/bookmarking. Just getting started right now, so there's not much there, but it should grow as I begin bookmarking/tagging interesting sites.

Turbogears, Django, and Subway

I'm still trying to get my mind into Python web programming. There are three main "pythonic megaframeworks" that I've seen get a lot of press lately, and trying to decide between them is a bit of a puzzle (as it always is in the Python web programming world). All three are seen, to some extent, as Ruby on Rails for Python.



My desire to get into Python web programming has to do with a little app I'm creating for businesses to track the time their employees/consultants spend at work and generate the appropriate payroll reports. So all of the descriptions below will be, to some degree, colored by this goal.

One framework I'm not really considering right now is Zope, partly because it seems to be "unpythonic." It's really a very nice system, especially for distributed teams of content builders. But I'm not such a team. Additionally, I've tried doing things with it before, and it seems powerful, but maybe a little too heavyweight for my current app. On the other hand, it's easy (and cheap) to get a Zope hosting account for your web app, whereas some of the other solutions may require dedicated servers.

Turbogears is the one I'm currently examining most closely. It basically ties together several pre-existing projects: CherryPy, Kid, SQLObject, MochiKit, and FormEncode being some of the major ones. It's coming along nicely, and is currently at version 0.8a5. What I'm really interested in here is the authentication and authorization framework they're adding in version 0.9. Not that it can't be used now; I'd just prefer not to roll my own identity module if possible.

Django is another framework that came out of the woodwork recently. I'm less familiar with it than with the others, though from what I've seen it is the most mature of the three. I mainly shy away from it because it seems that the developers have "reinvented the wheel" a bit. Rather than using some of the existing frameworks like CherryPy, they invented their own. Of course, those "existing frameworks" didn't exist (or weren't mature enough) when Django was started, so that's forgiveable. But I still like the idea of tying together other frameworks in a nice way rather than coding up something new.

Subway was created to directly compete with Rails (down to the name). It shares a lot in common with TurboGears. It integrates CherryPy and SQLObject, but where TurboGears uses Kid, Subway uses Cheetah templates. Subway was my first choice to begin the app (it seems to have appeared first of the three), but it doesn't seem to have the same developer momentum behind it. But it's worth following closely.

So there you have it. TurboGears, Django, and Subway. Each promises to radically simplify web development. The unfortunate thing for a Pythonisito like me is that "there's more than one way to do it." Doesn't sound too Pythonic, does it?

GRACE C to VHDL flow

For the MONARCH project, Georgia Tech is tasked with developing the compiler and some productivity utilities. Yesterday, I covered the compiler and the GUI. Today, I'll try to tackle the "rapid prototyping front-end."



For the prototyping environment, I've developed an internal representation, the "Human-Readable Internal Representation" (HumIR), with very Python-like syntax. Due to the need to target the hardware directly, I chose to make it statically typed (for now), but try not to force the user to declare things unnecessarily (so if X & Y are signed 32-bit integers, then Z = X+Y will create Z as a signed 32-bit integer). An example function which adds two streams of numbers is below:

 () = function add2streams():
   strm_id0 = 0
   strm_id1 = 1
   strm_id2 = 2
   while 1:
       x = mon_streamPopF(strm_id0)
       y = mon_streamPopF(strm_id1)
       mon_streamPush(strm_id2, x + y, 0)

Having developed the IR, I went on to create a flow that "lowers" the IR from the input syntax to the same format the Trimaran front-end uses to hand off to the back-end (the so-called "MONARCH Dataflow Graph", or MDFG for short.) Involved in this process is expression-lowering (so "z=(a+b) * c" becomes "t=a+b; z=t*c", for instance), if-conversion (removing all if/else blocks in the code by adding a "predicate" which can mask the execution of a statement), static single assignment (SSA) formation (which takes a predicated function, inserts select operations and makes all dataflow explicit). Finally, the resulting program is converted to a graph form and sent to the back-end.

The nice thing about this IR and flow (other than the fact that indentation as block structure really rocks) is that it was designed with hardware synthesis in mind (a dataflow machine is remarkably like custom-designed hardware). So I extended the flow with an instruction selection module, a modulo scheduler, and an automated pipelining module to generate nicely pipelined VHDL.

One other thing that's kind of nice about the flow is that HumIR is a high-level language, and as such, it's pretty easy to convert from (a currently very restricted subset of) C to HumIR. Given that I happen to have access to a C parser (EDG's parser is included in Trimaran), I went ahead and wrote a module (very alpha-level) that converts from so-called pcode (which has lisp-like syntax, nice and easy to parse) to HumIR. So now I have a two-headed (C and HumIR), two-tailed (MONARCH and VHDL) beast. With any luck, I'll have a tech report out soon which I can distribute freely, and maybe even some conference or journal papers.

Python in Compilers

OK, so this is my first post on my second blog. This one will be devoted to all the stuff I'm doing on Python. By way of introduction, I'll bring you up to speed on all the projects I'm working on right now. (Sorry, none are FOSS yet.) Today I'll focus on my full-time job.

My "real job" is a research engineer at Georgia Tech, working on the MONARCH project (part of the PCA project and the Morphware Forum). Our part of the project (at Georgia Tech) is to build a compiler ("Trimaran for XMONARCH", or trix for short) which targets the MONARCH embedded computer platform, containing a number of embedded RISC or RISC+SIMD (so-called "WideWord") CPUs and a large dataflow fabric. (MONARCH was designed jointly by Raytheon and the Information Sciences Institute at the University of Southern California.) Right now, we have a compiler that targets the dataflow portion of the chip, though we will be extending this to the more mundane RISC and SIMD parts. As such, I manage 4 grad students and do a lot of the coding myself. The dynamics of the project (defense applications and such) require that non-US citizens have limited access to the details of the chip, so I'm mainly focused on building the backend of the compiler.



Since I came on the project to build (from scratch) the backend of the compiler, and there was no need for binary or other compatibility with the Trimaran codebase (which we were using for the rest of the compiler), I decided to implement it in Python, which I had been dabbling with for several months. Having built a couple of code generators in the past in C++, I was rather pleased at the fact that Python increased my productivity 3-5x (by reducing both development time and lines of code produced). It also allows me to keep well ahead of the front-end development team (so I can manage them with all that extra time I saved! ;-) ).

Also as a part of the MONARCH project, we are developing a couple of productivity utilities. First, we have an Eclipse plugin which allows us to visualize the simulation of a program in the Raytheon-provided architecture simulator or any other dataflow-type simulator backend. The plugin communicates over sockets, so we have kind of a plugin-within-a-plugin, where our GUI plugs into Eclipse and the simulator plugs into our GUI. Most of the work here is Java (since we're using Eclipse), but the plug-in to Raytheon's simulator was written in Python, since its compatibility requirements were minor (command-line on the simulator side, XML over sockets on the GUI side). Again, quantum leap in productivity over statically, manifestly typed languages.

The second utility has really been a jumping-off point for my research. It's a "rapid prototyping language" for MONARCH dataflow programming, which I've extended to a full C to pipelined VHDL compiler. I'll give you more on this tomorrow.