Software Blag

In the Python programming language, metaclasses are used to specify how classes are created. Note that we are talking about class creation, not class instance creation. A few months ago I wrote some code which dynamically creates metaclasses. This turns out to be quite a useful trick.

Say you have a number of classes in a source file. To take an example from a project I am working on, each class opens and parses a particular file. Each class implements a common interface, and contains metadata which allows a manager to instantiate and “run” the class.

Of course, you want to run each class, and in the correct order. You might do something like this:

class Loader(object):
    [ ... ]

class RouteLoader(Loader):
    [ .... ]

class ShapeLoader(Loader):
    [ .... ]

loaders = [ RouteLoader, ShapeLoader ]

for loader in loaders:
    run(loader)

This sucks. Firstly, you are forced to manually maintain a list of loaders. Secondly, as you maintain the code, you might find that the order the loaders are defined in the file doesn’t match their order in loaders. I find the code much clearer if the order things run is the order they appear in the file; scrolling through the file shows you things in a logical order.

Instead, let’s define a function which creates a metaclass. This metaclass doesn’t do much, it just appends classes it constructs to a list:

def append_meta(lst):
    class AppendMeta(type):
        def __init__(cls, name, bases, dct):
            super(AppendMeta, cls).__init__(name, bases, dct)
            if hasattr(cls, "_ignore") and cls.__name__ in cls._ignore:
                return
            lst.append(cls)
    return AppendMeta

I’ll explain how this works in a second. Here’s how to use it:

loaders = []
mclass = append_meta(loaders)

class Loader(object):
    __metaclass__ = mclass
    _ignore = ('Loader', )
    [ .. ]

class RouteLoader(Loader):
    [ ... ]

class ShapeLoader(Loader):
    [ ... ]

for loader in loaders:
    run(loader)

What’s going on? append_meta is a function that takes a list reference as an argument, and returns a class. When used as a metaclass, it appends any class it creates to the provided list. The list reference is kept by the returned metaclass through normal closure behaviour. We have a little complexity to make sure that the Loader class itself doesn’t show up in the list, simply by providing a list of class names to ignore in class member _ignore. If someone can think of a more elegant way to do this, please let me know.

We no longer have to worry about maintaining the loaders list manually. As classes are created in the order they appear in the file, as we move them around in the file, the order they appear in the list is updated. Pretty neat.

I’m sure at this point you are wondering: why not just put all the loaders in a module, import it, and use dir(module)? Well, sometimes that doesn’t feel very nice; in any case, this is just an example. The approach of dynamically creating metaclasses to track and manipulate your classes is extremely useful. I have more complex examples that allow me to dynamically generate metaclasses which do several things, composing functionality. I’m using this to both track my classes and (using code pinched from Django’s ORM) also let certain class member variables know the name they have been given in the class. If there interest, I’ll blog about that - it is possible to come up with a library of useful operations performed by metaclasses, and then at runtime generate a metaclass to suit.

Anyway, that’s my first meaty post on this blog. Flame away!

For quite a while now my "personal" blog hasn't interested me much. I read other people's entries, and it's a good way to stay up to date with what people are doing, but I don't really care to post much other than photographs and rants. I'm going to start using this blog to publish lengthier entries on various things I've discovered while programming.

Anyway, that's the plan!

I've been working on a system that parses (and then interprets) Google Transit Data Feed information. My first shot at this used the ORM Storm (by the Canonical/Launchpad people). That wasn't a good fit, so I've rewritten it to use Django.

I think most projects should start up with a write, then immediate rewrite. It should almost be part of standard software development practise. I'm much happier with the code now. Printing out the timetable for any given stop is about twelve lines, and that includes the "logic" of figuring out what times buses will appear on any given day.

As this has been a spare time project, I've taken the liberty of absurdly over-engineering the whole thing. I've come up with some neat patterns for Python programs using metaclasses. Those ideas will be the subject of my next post, at which point (once people find and read this blog) I'm sure I'll get flamed.