uWSGI Transformations

Starting from uWSGI 1.9.7, a “transformations” API has been added to uWSGI internal routing.

A transformation is like a filter applied to the response generated by your application.

Transformations can be chained (the output of a transformation will be the input of the following one) and can completely overwriteresponse headers.

The most common example of transformation is gzip encoding. The output of your application is passed to a function compressing it with gzipand setting the Content-Encoding header. This feature rely on 2 external packages: libpcre3-dev, libz-dev on Ubuntu.

  1. [uwsgi]
  2. plugin = python,transformation_gzip
  3. http-socket = :9090
  4. ; load the werkzeug test app
  5. module = werkzeug.testapp:test_app
  6. ; if the client supports gzip encoding goto to the gzipper
  7. route-if = contains:${HTTP_ACCEPT_ENCODING};gzip goto:mygzipper
  8. route-run = last:
  9.  
  10. route-label = mygzipper
  11. ; pass the response to the gzip transformation
  12. route = ^/$ gzip:

The cachestore routing instruction is a transformation too, so you can cache various states of the response.

  1. [uwsgi]
  2. plugin = python,transformation_gzip
  3. http-socket = :9090
  4. ; load the werkezeug test app
  5. module = werkzeug.testapp:test_app
  6. ; create a cache of 100 items
  7. cache = 100
  8. ; if the client support gzip encoding goto to the gzipper
  9. route-if = contains:${HTTP_ACCEPT_ENCODING};gzip goto:mygzipper
  10. route = ^/$ cache:key=werkzeug_homepage
  11. route = ^/$ cachestore:key=werkzeug_homepage
  12. route-run = last:
  13.  
  14. route-label = mygzipper
  15. route = ^/$ cache:key=werkzeug_homepage.gz
  16. ; first cache the 'clean' response (for client not supporting gzip)
  17. route = ^/$ cachestore:key=werkzeug_homepage
  18. ; then pass the response to the gzip transformation
  19. route = ^/$ gzip:
  20. ; and cache it again in another item (gzipped)
  21. route = ^/$ cachestore:key=werkzeug_homepage.gz

Another common transformation is applying stylesheets to XML files. (see The XSLT plugin)

The toxslt transformation is exposed by the xslt plugin:

  1. uwsgi --plugin xslt --http-socket :9090 -w mycd --route-run "toxslt:stylesheet=t/xslt/cd.xml.xslt,params=foobar=test&agent=\${HTTP_USER_AGENT}"

The mycd module here is a simple XML generator. Its output is then passed to the XSLT transformation.

Streaming vs. buffering

Each transformation announces itself as a “streaming” one or a “buffering” one.

Streaming ones are transformations that can be applied to response chunks (parts). An example of a streaming transformationis gzip (you do not need the whole body to begin compressing it). Buffering transformations are those requiring the full body before applying something to it. XSLT is an example of buffering transformation. Another example of buffering transformations are those used for storing response in some kind of cache.

If your whole pipeline is composed by only “streaming” transformations, your client will receive the output chunk by chunk. On the other handa single buffering transformation will make the whole pipeline buffered, so your client will get the output only at the end.

An often using streaming functionality is gzip + chunked:

  1. [uwsgi]
  2. plugins = transformation_gzip,transformation_chunked
  3. route-run = gzip:
  4. route-run = chunked:
  5. ...

The whole transformation pipeline is composed by streaming plugins, so you will get each HTTP chunk in realtime.

Flushing magic

The “flush” transformation is a special one. It allows you to send the current contents of the transformation buffer to the client (without clearing the buffer).

You can use it for implementing streaming mode when buffering will be applied. A common example is having streaming + caching:

  1. [uwsgi]
  2. plugins = transformation_toupper,transform_tofile
  3. ; convert each char to uppercase
  4. route-run = toupper:
  5. ; after each chunk converted to upper case, flush to the client
  6. route-run = flush:
  7. ; buffer the whole response in memory for finally storing it in a file
  8. route-run = tofile:filename=/tmp/mycache
  9. ...

You can call flush multiple times and in various parts of the chain. Experiment a bit with it…

Available transformations (last update 20130504)

  • gzip, exposed by the transformation_gzip plugin (encode the response buffer to gzip)
  • toupper, exposed by the transformation_toupper plugin (example plugin transforming each character in uppercase)
  • tofile, exposed by the transformation_tofile plugin (used for caching to response buffer to a static file)
  • toxslt, exposed by the xslt plugin (apply xslt stylesheet to an XML response buffer)
  • cachestore, exposed by the router_cache plugin (cache the response buffer in the uWSGI cache)
  • chunked, encode the output in HTTP chunked
  • flush, flush the current buffer to the client
  • memcachedstore, store the response buffer in a memcached object
  • redisstore, store the response buffer in a redis object
  • template, apply routing translations to each chunk

Working on

  • rpc, allows applying rpc functions to a response buffer (limit 64k size)
  • lua, apply a lua function to a response buffer (no limit in size)