My misinterpretation of Python decorators

Posted 2008-06-22-Sun by qztxon
Categories: Python, Uncategorized

DrProject is written in Python, but I had only a little experience with Python when I first started. So I had to learn many things along the way, sometimes in the hard way.

Today’s topic is the semantics of decorators in Python (PEP 318). I used to think I knew how they worked, but I was wrong. Let’s start with a simple example:

def decorate(func):
    def decorated(self):
        print "Before"
        func(self)
        print "After"
    return decorated

class Foo(object):
    @decorate
    def bar(self):
        print "Hello, world!"

This is equivalent to the following code in the old style (I won’t repeat decorate because it remains unchanged):

class Foo(object):
    def bar(self):
        print "Hello, world!"
    bar = decorate(bar)

In either case (they’re equivalent), if we create a Foo and invoke bar, we get:

Before
Hello, world!
After

So far so good? This is the kind of code I found in DrProject:

def action(name, template=None, stylesheets=None):
    def _decorate(func):
        def _execute(controller, req):
            (...)
        (...)
        return _execute
    (...)
    return _decorate

class AdminController(Controller):
    @action('list_projects',
        template='admin_list_projects.html',
        stylesheets=['admin.css'])
    def _process_list_projects(self, req, order='name', desc=''):
        (... omitted ...)

Somehow, I naïvely expected the decorator invocation to be equivalent to something like _process_list_projects = action(_process_list_projects, 'list_projects', template='admin_list_projects.html', stylesheets=['admin.css']), with the function as the zeroth argument. But that would be impossible, because action only takes 3 arguments, and has no provisions for extra arguments – so Python would err loudly if action is called with 4 arguments.

The key to understanding this situation lies in PEP 318, but the code itself is very suggestive too. Essentially, _process_list_projects is decorated by the function returned by action('list_projects', template=..., stylesheets=...). (See how action returns a nested _decorate function?) So overall, the explicit form of this decoration would be as follows:

class AdminController(Controller):
    def _process_list_projects(self, req, order='name', desc=''):
        (... omitted ...)
    _process_list_projects = action('list_projects',
        template='admin_list_projects.html',
        stylesheets=['admin.css'])(_process_list_projects)

Notice the double call, i.e. action(...)(...).

To recap, a decorator usage like

@decorate
def foo():
    (...)

expands to the form

def foo():
    (...)
foo = decorate(foo)

where decorate is an arbitrary expression that evaluates to a decorator, which can be a simple function variable or the result of a function maker (e.g. make_decorator(1,2,3)).

I have nothing against the decorator syntax; in fact, I think it makes a lot of sense now that I understand it better. I just don’t want anyone else to repeat the same naïve misinterpretation.

A simplified overview of HTML forms

Posted 2008-06-22-Sun by qztxon
Categories: Uncategorized

A form is an element that can be added to an HTML page. It consists of named fields, with each field having a value. Through forms, a user can send information back to a web server.

Suppose we have a simple HTML form on the page alpha.php:

<form action="beta.php" method="post">
    <p>
        Enter your first name:
        <input type="text" name="firstname"/>
    </p>
    <p>
        Enter your last name:
        <input type="text" name="lastname"/>
    </p>
    <p><input type="submit"/></p>
</form>

This is what it looks like:

A simple HTML form

When the user John Smith comes along and submits this form, his web browser issues an HTTP POST request to the page beta.php with the following extra information: firstname = John, lastname = Smith. It should be apparent that the field names “firstname” and “lastname” are taken from the two input elements in the form.

For the technically inclined, the POST request looks like this in the literal Hypertext Transfer Protocol (unimportant details have been abridged):

POST /beta.php HTTP/1.1
Host: www.myserver.com
User-Agent: Mozilla Firefox 3.0
Accept: text/html
Content-Type: application/x-www-form-urlencoded
Content-Length: 29

firstname=John&lastname=Smith

As a preview for the next article, note that on most web server systems, alpha.php and beta.php would be in different files. And each page would be responsible for both handling the potential incoming POST data and generating the outgoing HTML page. But even though beta.php processes data from alpha.php, it does not refer to alpha.php – so it be hard to track down where beta.php is receiving POST data from. Also note that it is legal for a form to post back onto its own URL (in this case, change the form to action="alpha.php").

Miscellaneous humour pages

Posted 2008-06-17-Tue by qztxon
Categories: Uncategorized

More reading books

Posted 2008-06-16-Mon by qztxon
Categories: Uncategorized

These are the books that I’ve been reading lately:

Default behaviour of comparisons in Python

Posted 2008-06-09-Mon by qztxon
Categories: DrProject, Python, Uncategorized

Browsing DrProject’s code base a moment ago, I came across this piece of code in drproject.project.Project: (I abbreviated the long expression a bit)

    def get_members(self):
        members = [m.user for m in Membership.query.filter_by(...)]
        members.sort()
        return members

I was curious about what the sort actually did. So, members is a list of drproject.project.User objects. The class User extends Elixir’s Entity class. Neither of these classes appeared to have overridden the comparison operators. EntityMeta didn’t seem to do anything either. And the sort method was being called with no comparator function passed in. What on earth is going on?

The Python language reference page on comparisons was not very helpful. So I had to make some guesses. Since User objects are database rows, maybe they are sorted by their primary keys? But the aforementioned lack of code suggested against this possibility. Based on my experience with object-oriented languages, the only other thing I could think of was the IDs of objects — their memory addresses. It was a good candidate, since IDs are comparable for all objects.

A quick experiment in a Python session gives evidence for this behaviour:

In [1]: import random
In [2]: randcmp = lambda x, y: random.choice([-1, 0, +1])

In [3]: things = [object() for i in range(10)]
In [4]: things  # The IDs are in ascending order merely by coincidence.
Out[4]:
[<object object at 0xb7d6d598>,
 <object object at 0xb7d6d5a0>,
 <object object at 0xb7d6d5a8>,
 <object object at 0xb7d6d5b0>,
 <object object at 0xb7d6d5b8>,
 <object object at 0xb7d6d5c0>,
 <object object at 0xb7d6d5c8>,
 <object object at 0xb7d6d5d0>,
 <object object at 0xb7d6d5d8>,
 <object object at 0xb7d6d5e0>]

In [5]: things.sort(randcmp)
In [6]: things  # Now the IDs are in random order.
Out[6]:
[<object object at 0xb7d6d5c8>,
 <object object at 0xb7d6d5a8>,
 <object object at 0xb7d6d5d0>,
 <object object at 0xb7d6d598>,
 <object object at 0xb7d6d5a0>,
 <object object at 0xb7d6d5b0>,
 <object object at 0xb7d6d5d8>,
 <object object at 0xb7d6d5b8>,
 <object object at 0xb7d6d5c0>,
 <object object at 0xb7d6d5e0>]

In [7]: things.sort()
In [8]: things  # My gosh, we have ascending IDs again!
Out[8]:
[<object object at 0xb7d6d598>,
 <object object at 0xb7d6d5a0>,
 <object object at 0xb7d6d5a8>,
 <object object at 0xb7d6d5b0>,
 <object object at 0xb7d6d5b8>,
 <object object at 0xb7d6d5c0>,
 <object object at 0xb7d6d5c8>,
 <object object at 0xb7d6d5d0>,
 <object object at 0xb7d6d5d8>,
 <object object at 0xb7d6d5e0>]

Is this behaviour unintuitive, and does it let you shoot yourself in the foot easily? You be the judge.

And going back to the original DrProject code, the sort indeed sorts by IDs, which is probably not very useful.

Work plans

Posted 2008-06-06-Fri by qztxon
Categories: DrProject, Uncategorized

Four weeks have come and gone. I spent the first few weeks setting up software, familiarizing myself with the DrProject codebase (particularly its core components) and with SQLAlchemy and Elixir, and discussing with our HCI expert Liz Blankenship. Other than that, my major accomplishment to date is the successful creation of a web interface for editing roles and capabilities.

There’s still plenty of summer left, and certainly no shortage of tasks either:

  • Going for the big prize: Redesigning the web interface for creating and editing users and projects. Involves planning, designing, coding, and user testing. I forecast this will take about 5 weeks.
  • Ensure that everything doable in the web admin is also doable through XML RPC…? (I’ve heard some nasty things about our RPC framework, though.) I think this will take 3 weeks.
  • Bug fixes, small features/enhancements, and refactorings.
  • Fixing I/O or database performance issues. I’ll have to get more detail on that.
  • Sipping iced tea on the balcony while enjoying the cool spring breeze.

Roles and capabilities web editor

Posted 2008-06-04-Wed by qztxon
Categories: DrProject, Uncategorized

In goes a week of work, and presto: out comes the first working version of the web editor for roles and capabilities, being completed as of revision 4971 (changeset). Here is a screenshot for the visually inclined:

Edit roles and capabilities

Features:

  • Viewing the table (with real contents taken from the database, of course)
  • Adding a role based on an existing role
  • Updating the capabilities of a role
  • Removing roles
  • Detecting the addition of a role name that already exists
  • Detecting the addition of a blank role
  • Detection of a “based on” role that does not exist
  • A clean, easy-to-understand interface (but the table is really big)

To do:

  • Disable editing the “admin” and “empty” roles. Right now, no integrity is corrupted, but the error message is ugly and utterly incomprehensible.
  • When deleting a role, deal with users who still have that role in some project. Perhaps reset their role to “empty”.
  • Validate the names for new roles. ASCII-only? Length restrictions? Character restrictions (e.g. no space)?
  • Make empty or viewer as the default “based on”?
  • Make the names of the capabilities more user-friendly
  • Divide the two sections in a more visible way

Optimizing PNGs with optipng

Posted 2008-06-04-Wed by qztxon
Categories: Uncategorized

OptiPNG is a command-line program that losslessly recompresses PNG images. It does so by trying various zlib parameters (compression, memory, strategy, window size) and PNG filters (none, up, paeth, adaptive, etc.). It also reduces images with 256 or fewer colours to 8-bit paletted mode.

Here is an example of its operation:

me@computer:~$ optipng -zc 9 -zm 5-9 -zs 0 -f 0 "Edit Roles and Capabilities.png"
OptiPNG 0.5.5: Advanced PNG optimizer.
Copyright (C) 2001-2007 Cosmin Truta.

** Processing: Edit Roles and Capabilities.png
1264x1082 8-bit RGB-alpha non-interlaced
The image is losslessly reduced to 8-bit RGB
Input IDAT size = 132820 bytes
Input file size = 133140 bytes
Trying...
  zc = 9  zm = 9  zs = 0  f = 0		IDAT size = 113817
  zc = 9  zm = 8  zs = 0  f = 0		IDAT size = 113445
  zc = 9  zm = 7  zs = 0  f = 0		IDAT too big
  zc = 9  zm = 6  zs = 0  f = 0		IDAT too big
  zc = 9  zm = 5  zs = 0  f = 0		IDAT too big

Selecting parameters:
  zc = 9  zm = 8  zs = 0  f = 0		IDAT size = 113445

Output IDAT size = 113445 bytes (19375 bytes decrease)
Output file size = 113520 bytes (19620 bytes = 14.74% decrease)

Your results may vary. 0% to 20% size reduction is typical.

And one more thing: while pngcrush may have the better name, it is OptiPNG that has the better compression.

UI Ideas for Managing Roles and Capabilities

Posted 2008-05-30-Fri by qztxon
Categories: DrProject, Uncategorized

Liz and I have been busy discussing, and here’s what we’ve come up with initially:

DrProject web administration diagram

Posted 2008-05-28-Wed by qztxon
Categories: DrProject, Uncategorized

Here is the state diagram of the current DrProject web administration interface:
DrProject web administration diagram