Restful Web Services

REST stands for “REpresentational State Transfer” and it is a type of web service architecture and not, like SOAP, a protocol. In fact there is no standard for REST.

Loosely speaking REST says that a service can be thought of as a collection of resources. Each resource should be identified by a URL. There are four methods actions on a resource and they are POST (create), GET (read), PUT (update) and DELETE, from which the acronym CRUD (create-read-update-delete) stands for. A client communicates with the resource by making an HTTP request to the URL that identifies the resource and using the HTTP method POST/PUT/GET/DELETE to pass instructions to the resource. The URL may have an extension, for example json that specify how the protocol for encoding the data.

So for example a POST request to

http://127.0.0.1:8000/myapp/default/api/person

means that you want to create a new person. In this case a person may correspond to a record in table person but may also be some other type of resource (for example a file).

Similarly a GET request to

http://127.0.0.1:8000/myapp/default/api/persons.json

indicates a request for a list of persons (records from the data person) in json format.

A GET request to

http://127.0.0.1:8000/myapp/default/api/person/1.json

indicates a request for the information associated to person/1 (the record with id==1) and in json format.

In the case of web2py each request can be split into three parts:

• A first part that identify the location of the service, i.e. the action that exposes the service:

http://127.0.0.1:8000/myapp/default/api/

• The name of the resource (person, persons, person/1, etc.)
• The communication protocol specified by the extension.

Notice that we can always use the router to eliminate any unwanted prefix in the URL and for example simplify this:

http://127.0.0.1:8000/myapp/default/api/person/1.json

into this:

http://127.0.0.1:8000/api/person/1.json

yet this is a matter of taste and we have already discussed it at length in chapter 4.

In our example we used an action called api but this is not a requirement. We can in fact name the action that exposes the RESTful service any way we like and we can in fact even create more than one. For the sake of argument we will continue to assume that our RESTful action is called api.

We will also assume we have defined the following two tables:

db.define_table('person',
                Field('name'),
                Field('info'))
db.define_table('pet',
                Field('ownedby', db.person),
                Field('name'),
                Field('info'))

and they are the resources we want to expose.

The first thing we do is create the RESTful action:

def api():
    return locals()

Now we modify it so that the extension is filtered out of the request args (so that request.args can be used to identify the resource) and so that it can handle the different methods separately:

@request.restful()
def api():
    def GET(*args, **vars):
        return dict()
    def POST(*args, **vars):
        return dict()
    def PUT(*args, **vars):
        return dict()
    def DELETE(*args, **vars):
        return dict()
    return locals()

Now when we make a GET http request to

http://127.0.0.1:8000/myapp/default/api/person/1.json

it calls and returns GET('person','1') where GET is the function defined inside the action. Notice that:

• we do not need to define all four methods, only those that we wish to expose.
• the method function can take named arguments
• the extension is stored in request.extension and the content type is set automatically.

The @request.restful() decorator makes sure that the extension in the path info is stored into request.extension, maps the request method into the corresponding function within the action (POST, GET, PUT, DELETE), and passes request.args and request.vars to the selected function.

Now we build a service to POST and GET individual records:

@request.restful()
def api():
    response.view = 'generic.json'
    def GET(tablename, id):
        if not tablename == 'person':
            raise HTTP(400)
        return dict(person = db.person(id))
    def POST(tablename, **fields):
        if not tablename == 'person':
            raise HTTP(400)
        return db.person.validate_and_insert(**fields)
    return locals()

Notice that:

• the GET and POST are dealt with by different functions
• the function expect the correct arguments (un-named arguments parsed by request.args and named arguments are from request.vars)
• they check the input is correct and eventually raise an exception
• GET perform a select and returns the record, db.person(id). The output is automatically converted to JSON because the generic view is called.
• POST performs a validate_and_insert(..) and returns the id of the new record or, alternatively, validation errors. The POST variables, **fields, are the post variables.

parse_as_rest (experimental)

The logic explained so far is sufficient to create any type of RESTful web service yet web2py helps us even more.

In fact, web2py provides a syntax to describe which database tables we want to expose and how to map resource into URLs and vice versa.

This is done using URL patterns. A pattern is a string that maps the request args from a URL into a database query. There 4 types of atomic patterns:

• String constants for example “friend”
• String constant corresponding to a table. For example “friend[person]“ will match “friends” in the URL to the “person” table.
• Variables to be used to filter. For example “{person.id}” will apply a db.person.name=={person.id} filter.
• Names of fields, represented by “:field”

Atomic patterns can be combined into complex URL patterns using “/“ such as in

"/friend[person]/{person.id}/:field"

which gives a url of the form

http://..../friend/1/name

Into a query for a person.id that returns the name of the person. Here “friend[person]“ matches “friend” and filters the table “person”. “{person.id}” matches “1” and filters “person.id==1”. “:field” matches “name” and returns:

db(db.person.id==1).select().first().name

Multiple URL patters can be combined into a list so that one single RESTful action can serve different types of requests.

The DAL has a method parse_as_rest(pattern, args, vars) that given a list of patterns, the request.args and the request.vars matches the pattern and returns a response (GET only).

So here is a more complex example:

@request.restful()
def api():
    response.view = 'generic.' + request.extension
    def GET(*args, **vars):
        patterns = [
            "/friends[person]",
            "/friend/{person.name.startswith}",
            "/friend/{person.name}/:field",
            "/friend/{person.name}/pets[pet.ownedby]",
            "/friend/{person.name}/pet[pet.ownedby]/{pet.name}",
            "/friend/{person.name}/pet[pet.ownedby]/{pet.name}/:field"
            ]
        parser = db.parse_as_rest(patterns, args, vars)
        if parser.status == 200:
            return dict(content=parser.response)
        else:
            raise HTTP(parser.status, parser.error)
    def POST(table_name, **vars):
        if table_name == 'person':
            return dict(db.person.validate_and_insert(**vars))
        elif table_name == 'pet':
            return dict(db.pet.validate_and_insert(**vars))
        else:
            raise HTTP(400)
    return locals()

Which understands the following URLs that correspond to the listed patterns:

• GET all persons

http://.../api/friends

• GET one person with name starting with “t”

http://.../api/friend/t

• GET the “info” field value of the first person with name equal to “Tim”

http://.../api/friend/Tim/info

• GET a list of pets of the person (friend) above

http://.../api/friend/Tim/pets

• GET the pet with name “Snoopy of person with name “Tim”

http://.../api/friend/Tim/pet/Snoopy

• GET the “info” field value for the pet

http://.../api/friend/Tim/pet/Snoopy/info

The action also exposes two POST urls:

• POST a new friend
• POST a new pet

If you have the “curl” utility installed you can try:

$ curl -d "name=Tim" http://127.0.0.1:8000/myapp/default/api/friend.json
{"errors": {}, "id": 1}
$ curl http://127.0.0.1:8000/myapp/default/api/friends.json
{"content": [{"info": null, "name": "Tim", "id": 1}]}
$ curl -d "name=Snoopy&ownedby=1" http://127.0.0.1:8000/myapp/default/api/pet.json
{"errors": {}, "id": 1}
$ curl http://127.0.0.1:8000/myapp/default/api/friend/Tim/pet/Snoopy.json
{"content": [{"info": null, "ownedby": 1, "name": "Snoopy", "id": 1}]}

It is possible to declare more complex queries such where a value in the URL is used to build a query not involving equality. For example

patterns = ['friends/{person.name.contains}'

maps

http://..../friends/i

into

db.person.name.contains('i')

And similarly:

patterns = ['friends/{person.name.ge}/{person.name.gt.not}'

maps

http://..../friends/aa/uu

into

(db.person.name>='aa') & (~(db.person.name>'uu'))

valid attributes for a field in a pattern are: contains, startswith, le, ge, lt, gt, eq (equal, default), ne (not equal). Other attributes specifically for date and datetime fields are day, month, year, hour, minute, second.

Notice that this pattern syntax is not designed to be general. Not every possible query can be described via a pattern but a lot of them are. The syntax may be extended in the future.

Often you want to expose some RESTful URLs but you want to restrict the possible queries. This can be done by passing an extra argument queries to the parse_as_rest method. queries is a dictionary of (tablename,query) where query is a DAL query to restrict access to table tablename.

We can also order results using the order GET variables

http://..../api/friends?order=name|~info

which order alphabetically (name) and then by reversed info order.

We can also limit the number of records by specifying a limit and offset GET variables

http://..../api/friends?offset=10&limit=1000

which will return up to 1000 friends (persons) and skip the first 10. limit defaults to 1000 and offset default to 0.

Let’s now consider an extreme case. We want to build all possible patterns for all tables (except auth_ tables). We want to be able to search by any text field, any integer field, any double field (by range) and any date (also by range). We also want to be able to POST into any table:

In the general case this requires a lot of patterns. Web2py makes it simple:

@request.restful()
def api():
    response.view = 'generic.' + request.extension
    def GET(*args, **vars):
        patterns = 'auto'
        parser = db.parse_as_rest(patterns, args, vars)
        if parser.status == 200:
            return dict(content=parser.response)
        else:
            raise HTTP(parser.status, parser.error)
    def POST(table_name, **vars):
        return dict(db[table_name].validate_and_insert(**vars))
    return locals()

Settings patterns='auto' results in web2py generating all possible patterns for all non-auth tables. There is even a pattern for querying about patterns:

http://..../api/patterns.json

which for out person and pet tables results in:

{"content": [
   "/person[person]",
   "/person/id/{person.id}",
   "/person/id/{person.id}/:field",
   "/person/id/{person.id}/pet[pet.ownedby]",
   "/person/id/{person.id}/pet[pet.ownedby]/id/{pet.id}",
   "/person/id/{person.id}/pet[pet.ownedby]/id/{pet.id}/:field",
   "/person/id/{person.id}/pet[pet.ownedby]/ownedby/{pet.ownedby}",
   "/person/id/{person.id}/pet[pet.ownedby]/ownedby/{pet.ownedby}/:field",
   "/person/name/pet[pet.ownedby]",
   "/person/name/pet[pet.ownedby]/id/{pet.id}",
   "/person/name/pet[pet.ownedby]/id/{pet.id}/:field",
   "/person/name/pet[pet.ownedby]/ownedby/{pet.ownedby}",
   "/person/name/pet[pet.ownedby]/ownedby/{pet.ownedby}/:field",
   "/person/info/pet[pet.ownedby]",
   "/person/info/pet[pet.ownedby]/id/{pet.id}",
   "/person/info/pet[pet.ownedby]/id/{pet.id}/:field",
   "/person/info/pet[pet.ownedby]/ownedby/{pet.ownedby}",
   "/person/info/pet[pet.ownedby]/ownedby/{pet.ownedby}/:field",
   "/pet[pet]",
   "/pet/id/{pet.id}",
   "/pet/id/{pet.id}/:field",
   "/pet/ownedby/{pet.ownedby}",
   "/pet/ownedby/{pet.ownedby}/:field"
]}

You can specify auto patterns for some tables only:

patterns = [':auto[person]',':auto[pet]']

smart_query (experimental)

There are times when you need more flexibility and you want to be able to pass to a RESTful service an arbitrary query like

http://.../api.json?search=person.name starts with 'T' and person.name contains 'm'

You can do this using

@request.restful()
def api():
    response.view = 'generic.' + request.extension
    def GET(search):
        try:
            rows = db.smart_query([db.person, db.pet], search).select()
            return dict(result=rows)
        except RuntimeError:
            raise HTTP(400, "Invalid search string")
    def POST(table_name, **vars):
        return dict(db[table_name].validate_and_insert(**vars))
    return locals()

The method db.smart_query takes two arguments:

• a list of field or table that should be allowed in the query
• a string containing the query expressed in natural language

and it returns a db.set object with the records that have been found.

Notice that the search string is parsed, not evaluated or executed and therefore it provides no security risk.

Access Control

Access to the API can be restricted as usual by using decorators. So, for example

auth.settings.allow_basic_login = True
@auth.requires_login()
@request.restful()
def api():
   def GET(s):
       return 'access granted, you said %s' % s
   return locals()

can now be accessed with

$ curl --user name:password http://127.0.0.1:8000/myapp/default/api/hello
access granted, you said hello