Comment Example

Introduction

This is an example of how to write WSGI middleware with WebOb. The specific example adds a simple comment form to HTML web pages; any page served through the middleware that is HTML gets a comment form added to it, and shows any existing comments.

Code

The finished code for this is available in docs/comment-example-code/example.py — you can run that file as a script to try it out.

Instantiating Middleware

Middleware of any complexity at all is usually best created as a class with its configuration as arguments to that class.

Every middleware needs an application (app) that it wraps. This middleware also needs a location to store the comments; we’ll put them all in a single directory.

import os
class Commenter(object):
    def __init__(self, app, storage_dir):
        self.app = app
        self.storage_dir = storage_dir
        if not os.path.exists(storage_dir):
            os.makedirs(storage_dir)

When you use this middleware, you’ll use it like:

app = ... make the application ...
app = Commenter(app, storage_dir='./comments')

For our application we’ll use a simple static file server that is included with Paste (use easy_install Paste to install this). The setup is all at the bottom of example.py, and looks like this:

if __name__ == '__main__':
    import optparse
    parser = optparse.OptionParser(
        usage='%prog --port=PORT BASE_DIRECTORY'
        )
    parser.add_option(
        '-p', '--port',
        default='8080',
        dest='port',
        type='int',
        help='Port to serve on (default 8080)')
    parser.add_option(
        '--comment-data',
        default='./comments',
        dest='comment_data',
        help='Place to put comment data into (default ./comments/)')
    options, args = parser.parse_args()
    if not args:
        parser.error('You must give a BASE_DIRECTORY')
    base_dir = args[0]
    from paste.urlparser import StaticURLParser
    app = StaticURLParser(base_dir)
    app = Commenter(app, options.comment_data)
    from wsgiref.simple_server import make_server
    httpd = make_server('localhost', options.port, app)
    print 'Serving on http://localhost:%s' % options.port
    try:
        httpd.serve_forever()
    except KeyboardInterrupt:
        print '^C'

I won’t explain it here, but basically it takes some options, creates an application that serves static files (StaticURLParser(base_dir)), wraps it with Commenter(app, options.comment_data) then serves that.

The Middleware

While we’ve created the class structure for the middleware, it doesn’t actually do anything. Here’s a kind of minimal version of the middleware (using WebOb):

from webob import Request
class Commenter(object):
    def __init__(self, app, storage_dir):
        self.app = app
        self.storage_dir = storage_dir
        if not os.path.exists(storage_dir):
            os.makedirs(storage_dir)
    def __call__(self, environ, start_response):
        req = Request(environ)
        resp = req.get_response(self.app)
        return resp(environ, start_response)

This doesn’t modify the response it any way. You could write it like this without WebOb:

class Commenter(object):
    ...
    def __call__(self, environ, start_response):
        return self.app(environ, start_response)

But it won’t be as convenient later. First, lets create a little bit of infrastructure for our middleware. We need to save and load per-url data (the comments themselves). We’ll keep them in pickles, where each url has a pickle named after the url (but double-quoted, so http://localhost:8080/index.html becomes http%3A%2F%2Flocalhost%3A8080%2Findex.html).

from cPickle import load, dump
class Commenter(object):
    ...
    def get_data(self, url):
        filename = self.url_filename(url)
        if not os.path.exists(filename):
            return []
        else:
            f = open(filename, 'rb')
            data = load(f)
            f.close()
            return data
    def save_data(self, url, data):
        filename = self.url_filename(url)
        f = open(filename, 'wb')
        dump(data, f)
        f.close()
    def url_filename(self, url):
        # Double-quoting makes the filename safe
        return os.path.join(self.storage_dir, urllib.quote(url, ''))

You can get the full request URL with req.url, so to get the comment data with these methods you do data = self.get_data(req.url).

Now we’ll update the __call__ method to filter some responses, and get the comment data for those. We don’t want to change responses that were error responses (anything but 200), nor do we want to filter responses that aren’t HTML. So we get:

class Commenter(object):
    ...
    def __call__(self, environ, start_response):
        req = Request(environ)
        resp = req.get_response(self.app)
        if resp.content_type != 'text/html' or resp.status_code != 200:
            return resp(environ, start_response)
        data = self.get_data(req.url)
        ... do stuff with data, update resp ...
        return resp(environ, start_response)

So far we’re punting on actually adding the comments to the page. We also haven’t defined what data will hold. Let’s say it’s a list of dictionaries, where each dictionary looks like {'name': 'John Doe', 'homepage': 'http://blog.johndoe.com', 'comments': 'Great site!'}.

We’ll also need a simple method to add stuff to the page. We’ll use a regular expression to find the end of the page and put text in:

import re
class Commenter(object):
    ...
    _end_body_re = re.compile(r'</body.*?>', re.I|re.S)
    def add_to_end(self, html, extra_html):
        """
        Adds extra_html to the end of the html page (before </body>)
        """
        match = self._end_body_re.search(html)
        if not match:
            return html + extra_html
        else:
            return html[:match.start()] + extra_html + html[match.start():]

And then we’ll use it like:

data = self.get_data(req.url)
body = resp.body
body = self.add_to_end(body, self.format_comments(data))
resp.body = body
return resp(environ, start_response)

We get the body, update it, and put it back in the response. This also updates Content-Length. Then we define:

from webob import html_escape
class Commenter(object):
    ...
    def format_comments(self, comments):
        if not comments:
            return ''
        text = []
        text.append('<hr>')
        text.append('<h2><a name="comment-area"></a>Comments (%s):</h2>' % len(comments))
        for comment in comments:
            text.append('<h3><a href="%s">%s</a> at %s:</h3>' % (
                html_escape(comment['homepage']), html_escape(comment['name']),
                time.strftime('%c', comment['time'])))
            # Susceptible to XSS attacks!:
            text.append(comment['comments'])
        return ''.join(text)

We put in a header (with an anchor we’ll use later), and a section for each comment. Note that html_escape is the same as cgi.escape and just turns & into &, etc.

Because we put in some text without quoting it is susceptible to a Cross-Site Scripting attack. Fixing that is beyond the scope of this tutorial; you could quote it or clean it with something like lxml.html.clean.

Accepting Comments

All of those pieces display comments, but still no one can actually make comments. To handle this we’ll take a little piece of the URL space for our own, everything under /.comments, so when someone POSTs there it will add a comment.

When the request comes in there are two parts to the path: SCRIPT_NAME and PATH_INFO. Everything in SCRIPT_NAME has already been parsed, and everything in PATH_INFO has yet to be parsed. That means that the URL without PATH_INFO is the path to the middleware; we can intercept anything else below SCRIPT_NAME but nothing above it. The name for the URL without PATH_INFO is req.application_url. We have to capture it early to make sure it doesn’t change (since the WSGI application we are wrapping may update SCRIPT_NAME and PATH_INFO).

So here’s what this all looks like:

class Commenter(object):
    ...
    def __call__(self, environ, start_response):
        req = Request(environ)
        if req.path_info_peek() == '.comments':
            return self.process_comment(req)(environ, start_response)
        # This is the base path of *this* middleware:
        base_url = req.application_url
        resp = req.get_response(self.app)
        if resp.content_type != 'text/html' or resp.status_code != 200:
            # Not an HTML response, we don't want to
            # do anything to it
            return resp(environ, start_response)
        # Make sure the content isn't gzipped:
        resp.decode_content()
        comments = self.get_data(req.url)
        body = resp.body
        body = self.add_to_end(body, self.format_comments(comments))
        body = self.add_to_end(body, self.submit_form(base_url, req))
        resp.body = body
        return resp(environ, start_response)

base_url is the path where the middleware is located (if you run the example server, it will be http://localhost:PORT/). We use req.path_info_peek() to look at the next segment of the URL — what comes after base_url. If it is .comments then we handle it internally and don’t pass the request on.

We also put in a little guard, resp.decode_content() in case the application returns a gzipped response.

Then we get the data, add the comments, add the form to make new comments, and return the result.

submit_form

Here’s what the form looks like:

class Commenter(object):
    ...
    def submit_form(self, base_path, req):
        return '''<h2>Leave a comment:</h2>
        <form action="%s/.comments" method="POST">
         <input type="hidden" name="url" value="%s">
         <table width="100%%">
          <tr><td>Name:</td>
              <td><input type="text" name="name" style="width: 100%%"></td></tr>
          <tr><td>URL:</td>
              <td><input type="text" name="homepage" style="width: 100%%"></td></tr>
         </table>
         Comments:<br>
         <textarea name="comments" rows=10 style="width: 100%%"></textarea><br>
         <input type="submit" value="Submit comment">
        </form>
        ''' % (base_path, html_escape(req.url))

Nothing too exciting. It submits a form with the keys url (the URL being commented on), name, homepage, and comments.

process_comment

If you look at the method call, what we do is call the method then treat the result as a WSGI application:

return self.process_comment(req)(environ, start_response)

You could write this as:

response = self.process_comment(req)
return response(environ, start_response)

A common pattern in WSGI middleware that doesn’t use WebOb is to just do:

return self.process_comment(environ, start_response)

But the WebOb style makes it easier to modify the response if you want to; modifying a traditional WSGI response/application output requires changing your logic flow considerably.

Here’s the actual processing code:

from webob import exc
from webob import Response
class Commenter(object):
    ...
    def process_comment(self, req):
        try:
            url = req.params['url']
            name = req.params['name']
            homepage = req.params['homepage']
            comments = req.params['comments']
        except KeyError, e:
            resp = exc.HTTPBadRequest('Missing parameter: %s' % e)
            return resp
        data = self.get_data(url)
        data.append(dict(
            name=name,
            homepage=homepage,
            comments=comments,
            time=time.gmtime()))
        self.save_data(url, data)
        resp = exc.HTTPSeeOther(location=url+'#comment-area')
        return resp

We either give a Bad Request response (if the form submission is somehow malformed), or a redirect back to the original page.

The classes in webob.exc (like HTTPBadRequest and HTTPSeeOther) are Response subclasses that can be used to quickly create responses for these non-200 cases where the response body usually doesn’t matter much.

Conclusion

This shows how to make response modifying middleware, which is probably the most difficult kind of middleware to write with WSGI — modifying the request is quite simple in comparison, as you simply update environ.