JSON-RPC Example

Introduction

This is an example of how to write a web service using WebOb. The example shows how to create a JSON-RPC endpoint using WebOb and the simplejson JSON library. This also shows how to use WebOb as a client library using WSGIProxy.

While this example presents JSON-RPC, this is not an endorsement of JSON-RPC. In fact I don’t like JSON-RPC. It’s unnecessarily un-RESTful, and modelled too closely on XML-RPC.

Code

The finished code for this is available in docs/json-example-code/jsonrpc.py — you can run that file as a script to try it out, or import it.

Concepts

JSON-RPC wraps an object, allowing you to call methods on that object and get the return values. It also provides a way to get error responses.

The specification goes into the details (though in a vague sort of way). Here’s the basics:

• All access goes through a POST to a single URL.

• The POST contains a JSON body that looks like:

  {"method": "methodName",
   "id": "arbitrary-something",
   "params": [arg1, arg2, ...]}

• The id parameter is just a convenience for the client to keep track of which response goes with which request. This makes asynchronous calls (like an XMLHttpRequest) easier. We just send the exact same id back as we get, we never look at it.

• The response is JSON. A successful response looks like:

  {"result": the_result,
   "error": null,
   "id": "arbitrary-something"}

• The error response looks like:

  {"result": null,
   "error": {"name": "JSONRPCError",
             "code": (number 100-999),
             "message": "Some Error Occurred",
             "error": "whatever you want\n(a traceback?)"},
   "id": "arbitrary-something"}

• It doesn’t seem to indicate if an error response should have a 200 response or a 500 response. So as not to be completely stupid about HTTP, we choose a 500 resonse, as giving an error with a 200 response is irresponsible.

Infrastructure

To make this easier to test, we’ll set up a bit of infrastructure. This will open up a server (using wsgiref) and serve up our application (note that creating the application is left out to start with):

import sys
def main(args=None):
    import optparse
    from wsgiref import simple_server
    parser = optparse.OptionParser(
        usage="%prog [OPTIONS] MODULE:EXPRESSION")
    parser.add_option(
        '-p', '--port', default='8080',
        help='Port to serve on (default 8080)')
    parser.add_option(
        '-H', '--host', default='127.0.0.1',
        help='Host to serve on (default localhost; 0.0.0.0 to make public)')
    if args is None:
        args = sys.argv[1:]
    options, args = parser.parse_args()
    if not args or len(args) > 1:
        print 'You must give a single object reference'
        parser.print_help()
        sys.exit(2)
    app = make_app(args[0])
    server = simple_server.make_server(
        options.host, int(options.port),
        app)
    print 'Serving on http://%s:%s' % (options.host, options.port)
    server.serve_forever()
if __name__ == '__main__':
    main()

I won’t describe this much. It starts a server, serving up just the app created by make_app(args[0]). make_app will have to load up the object and wrap it in our WSGI/WebOb wrapper. We’ll be calling that wrapper JSONRPC(obj), so here’s how it’ll go:

def make_app(expr):
    module, expression = expr.split(':', 1)
    __import__(module)
    module = sys.modules[module]
    obj = eval(expression, module.__dict__)
    return JsonRpcApp(obj)

We use __import__(module) to import the module, but its return value is wonky. We can find the thing it imported in sys.modules (a dictionary of all the loaded modules). Then we evaluate the second part of the expression in the namespace of the module. This lets you do something like smtplib:SMTP('localhost') to get a fully instantiated SMTP object.

That’s all the infrastructure we’ll need for the server side. Now we just have to implement JsonRpcApp.

The Application Wrapper

Note that I’m calling this an “application” because that’s the terminology WSGI uses. Everything that gets called is an “application”, and anything that calls an application is called a “server”.

The instantiation of the server is already figured out:

class JsonRpcApp(object):
    def __init__(self, obj):
        self.obj = obj
    def __call__(self, environ, start_response):
        ... the WSGI interface ...

So the server is an instance bound to the particular object being exposed, and __call__ implements the WSGI interface.

We’ll start with a simple outline of the WSGI interface, using a kind of standard WebOb setup:

from webob import Request, Response
from webob import exc
class JsonRpcApp(object):
    ...
    def __call__(self, environ, start_response):
        req = Request(environ)
        try:
            resp = self.process(req)
        except ValueError, e:
            resp = exc.HTTPBadRequest(str(e))
        except exc.HTTPException, e:
            resp = e
        return resp(environ, start_response)

We first create a request object. The request object just wraps the WSGI environment. Then we create the response object in the process method (which we still have to write). We also do some exception catching. We’ll turn any ValueError into a 400 Bad Request response. We’ll also let process raise any web.exc.HTTPException exception. There’s an exception defined in that module for all the HTTP error responses, like 405 Method Not Allowed. These exceptions are themselves WSGI applications (as is webob.Response), and so we call them like WSGI applications and return the result.

The process method

The process method of course is where all the fancy stuff happens. We’ll start with just the most minimal implementation, with no error checking or handling:

from simplejson import loads, dumps
class JsonRpcApp(object):
    ...
    def process(self, req):
        json = loads(req.body)
        method = json['method']
        params = json['params']
        id = json['id']
        method = getattr(self.obj, method)
        result = method(*params)
        resp = Response(
            content_type='application/json',
            body=dumps(dict(result=result,
                            error=None,
                            id=id)))
        return resp

As long as the request is properly formed and the method doesn’t raise any exceptions, you are pretty much set. But of course that’s not a reasonable expectation. There’s a whole bunch of things that can go wrong. For instance, it has to be a POST method:

if not req.method == 'POST':
    raise exc.HTTPMethodNotAllowed(
        "Only POST allowed",
        allowed='POST')

And maybe the request body doesn’t contain valid JSON:

try:
    json = loads(req.body)
except ValueError, e:
    raise ValueError('Bad JSON: %s' % e)

And maybe all the keys aren’t in the dictionary:

try:
    method = json['method']
    params = json['params']
    id = json['id']
except KeyError, e:
    raise ValueError(
        "JSON body missing parameter: %s" % e)

And maybe it’s trying to acces a private method (a method that starts with _) — that’s not just a bad request, we’ll call that case 403 Forbidden.

if method.startswith('_'):
    raise exc.HTTPForbidden(
        "Bad method name %s: must not start with _" % method)

And maybe json['params'] isn’t a list:

if not isinstance(params, list):
    raise ValueError(
        "Bad params %r: must be a list" % params)

And maybe the method doesn’t exist:

try:
    method = getattr(self.obj, method)
except AttributeError:
    raise ValueError(
        "No such method %s" % method)

The last case is the error we actually can expect: that the method raises some exception.

try:
    result = method(*params)
except:
    tb = traceback.format_exc()
    exc_value = sys.exc_info()[1]
    error_value = dict(
        name='JSONRPCError',
        code=100,
        message=str(exc_value),
        error=tb)
    return Response(
        status=500,
        content_type='application/json',
        body=dumps(dict(result=None,
                        error=error_value,
                        id=id)))

That’s a complete server.

The Complete Code

Since we showed all the error handling in pieces, here’s the complete code:

from webob import Request, Response
from webob import exc
from simplejson import loads, dumps
import traceback
import sys
class JsonRpcApp(object):
    """
    Serve the given object via json-rpc (http://json-rpc.org/)
    """
    def __init__(self, obj):
        self.obj = obj
    def __call__(self, environ, start_response):
        req = Request(environ)
        try:
            resp = self.process(req)
        except ValueError, e:
            resp = exc.HTTPBadRequest(str(e))
        except exc.HTTPException, e:
            resp = e
        return resp(environ, start_response)
    def process(self, req):
        if not req.method == 'POST':
            raise exc.HTTPMethodNotAllowed(
                "Only POST allowed",
                allowed='POST')
        try:
            json = loads(req.body)
        except ValueError, e:
            raise ValueError('Bad JSON: %s' % e)
        try:
            method = json['method']
            params = json['params']
            id = json['id']
        except KeyError, e:
            raise ValueError(
                "JSON body missing parameter: %s" % e)
        if method.startswith('_'):
            raise exc.HTTPForbidden(
                "Bad method name %s: must not start with _" % method)
        if not isinstance(params, list):
            raise ValueError(
                "Bad params %r: must be a list" % params)
        try:
            method = getattr(self.obj, method)
        except AttributeError:
            raise ValueError(
                "No such method %s" % method)
        try:
            result = method(*params)
        except:
            text = traceback.format_exc()
            exc_value = sys.exc_info()[1]
            error_value = dict(
                name='JSONRPCError',
                code=100,
                message=str(exc_value),
                error=text)
            return Response(
                status=500,
                content_type='application/json',
                body=dumps(dict(result=None,
                                error=error_value,
                                id=id)))
        return Response(
            content_type='application/json',
            body=dumps(dict(result=result,
                            error=None,
                            id=id)))

The Client

It would be nice to have a client to test out our server. Using WSGIProxy we can use WebOb Request and Response to do actual HTTP connections.

The basic idea is that you can create a blank Request:

>>> from webob import Request
>>> req = Request.blank('http://python.org')

Then you can send that request to an application:

>>> from wsgiproxy.exactproxy import proxy_exact_request
>>> resp = req.get_response(proxy_exact_request)

This particular application (proxy_exact_request) sends the request over HTTP:

>>> resp.content_type
'text/html'
>>> resp.body[:10]
'<!DOCTYPE '

So we’re going to create a proxy object that constructs WebOb-based jsonrpc requests, and sends those using proxy_exact_request.

The Proxy Client

The proxy client is instantiated with its base URL. We’ll also let you pass in a proxy application, in case you want to do local requests (e.g., to do direct tests against a JsonRpcApp instance):

class ServerProxy(object):
    def __init__(self, url, proxy=None):
        self._url = url
        if proxy is None:
            from wsgiproxy.exactproxy import proxy_exact_request
            proxy = proxy_exact_request
        self.proxy = proxy

This ServerProxy object itself doesn’t do much, but you can call methods on it. We can intercept any access ServerProxy(...).method with the magic function __getattr__. Whenever you get an attribute that doesn’t exist in an instance, Python will call inst.__getattr__(attr_name) and return that. When you call a method, you are calling the object that .method returns. So we’ll create a helper object that is callable, and our __getattr__ will just return that:

class ServerProxy(object):
    ...
    def __getattr__(self, name):
        # Note, even attributes like __contains__ can get routed
        # through __getattr__
        if name.startswith('_'):
            raise AttributeError(name)
        return _Method(self, name)
class _Method(object):
    def __init__(self, parent, name):
        self.parent = parent
        self.name = name

Now when we call the method we’ll be calling _Method.__call__, and the HTTP endpoint will be self.parent._url, and the method name will be self.name.

Here’s the code to do the call:

class _Method(object):
    ...
    def __call__(self, *args):
        json = dict(method=self.name,
                    id=None,
                    params=list(args))
        req = Request.blank(self.parent._url)
        req.method = 'POST'
        req.content_type = 'application/json'
        req.body = dumps(json)
        resp = req.get_response(self.parent.proxy)
        if resp.status_code != 200 and not (
            resp.status_code == 500
            and resp.content_type == 'application/json'):
            raise ProxyError(
                "Error from JSON-RPC client %s: %s"
                % (self._url, resp.status),
                resp)
        json = loads(resp.body)
        if json.get('error') is not None:
            e = Fault(
                json['error'].get('message'),
                json['error'].get('code'),
                json['error'].get('error'),
                resp)
            raise e
        return json['result']

We raise two kinds of exceptions here. ProxyError is when something unexpected happens, like a 404 Not Found. Fault is when a more expected exception occurs, i.e., the underlying method raised an exception.

In both cases we’ll keep the response object around, as that can be interesting. Note that you can make exceptions have any methods or signature you want, which we’ll do:

class ProxyError(Exception):
    """
    Raised when a request via ServerProxy breaks
    """
    def __init__(self, message, response):
        Exception.__init__(self, message)
        self.response = response
class Fault(Exception):
    """
    Raised when there is a remote error
    """
    def __init__(self, message, code, error, response):
        Exception.__init__(self, message)
        self.code = code
        self.error = error
        self.response = response
    def __str__(self):
        return 'Method error calling %s: %s\n%s' % (
            self.response.request.url,
            self.args[0],
            self.error)

Using Them Together

Good programmers start with tests. But at least we’ll end with a test. We’ll use doctest for our tests. The test is in docs/json-example-code/test_jsonrpc.txt and you can run it with docs/json-example-code/test_jsonrpc.py, which looks like:

if __name__ == '__main__':
    import doctest
    doctest.testfile('test_jsonrpc.txt')

As you can see, it’s just a stub to run the doctest. We’ll need a simple object to expose. We’ll make it real simple:

>>> class Divider(object):
...     def divide(self, a, b):
...        return a / b

Then we’ll get the app setup:

>>> from jsonrpc import *
>>> app = JsonRpcApp(Divider())

And attach the client directly to it:

>>> proxy = ServerProxy('http://localhost:8080', proxy=app)

Because we gave the app itself as the proxy, the URL doesn’t actually matter.

Now, if you are used to testing you might ask: is this kosher? That is, we are shortcircuiting HTTP entirely. Is this a realistic test?

One thing you might be worried about in this case is that there are more shared objects than you’d have with HTTP. That is, everything over HTTP is serialized to headers and bodies. Without HTTP, we can send stuff around that can’t go over HTTP. This could happen, but we’re mostly protected because the only thing the application’s share is the WSGI environ. Even though we use a webob.Request object on both side, it’s not the same request object, and all the state is studiously kept in the environment. We could share things in the environment that couldn’t go over HTTP. For instance, we could set environ['jsonrpc.request_value'] = dict(...), and avoid simplejson.dumps and simplejson.loads. We could do that, and if we did then it is possible our test would work even though the libraries were broken over HTTP. But of course inspection shows we don’t do that. A little discipline is required to resist playing clever tricks (or else you can play those tricks and do more testing). Generally it works well.

So, now we have a proxy, lets use it:

>>> proxy.divide(10, 4)
2
>>> proxy.divide(10, 4.0)
2.5

Lastly, we’ll test a couple error conditions. First a method error:

>>> proxy.divide(10, 0) 
Traceback (most recent call last):
   ...
Fault: Method error calling http://localhost:8080: integer division or modulo by zero
Traceback (most recent call last):
  File ...
    result = method(*params)
  File ...
    return a / b
ZeroDivisionError: integer division or modulo by zero

It’s hard to actually predict this exception, because the test of the exception itself contains the traceback from the underlying call, with filenames and line numbers that aren’t stable. We use # doctest: +ELLIPSIS so that we can replace text we don’t care about with .... This is actually figured out through copy-and-paste, and visual inspection to make sure it looks sensible.

The other exception can be:

>>> proxy.add(1, 1)
Traceback (most recent call last):
    ...
ProxyError: Error from JSON-RPC client http://localhost:8080: 400 Bad Request

Here the exception isn’t a JSON-RPC method exception, but a more basic ProxyError exception.

Conclusion

Hopefully this will give you ideas about how to implement web services of different kinds using WebOb. I hope you also can appreciate the elegance of the symmetry of the request and response objects, and the client and server for the protocol.

Many of these techniques would be better used with a RESTful service, so do think about that direction if you are implementing your own protocol.