Unicode in Flask

Flask, like Jinja2 and Werkzeug, is totally Unicode based when it comes totext. Not only these libraries, also the majority of web related Pythonlibraries that deal with text. If you don’t know Unicode so far, youshould probably read The Absolute Minimum Every Software DeveloperAbsolutely, Positively Must Know About Unicode and Character Sets.This part of the documentation just tries to cover the very basics sothat you have a pleasant experience with Unicode related things.

Automatic Conversion

Flask has a few assumptions about your application (which you can changeof course) that give you basic and painless Unicode support:

  • the encoding for text on your website is UTF-8

  • internally you will always use Unicode exclusively for text exceptfor literal strings with only ASCII character points.

  • encoding and decoding happens whenever you are talking over a protocolthat requires bytes to be transmitted.

So what does this mean to you?

HTTP is based on bytes. Not only the protocol, also the system used toaddress documents on servers (so called URIs or URLs). However HTML whichis usually transmitted on top of HTTP supports a large variety ofcharacter sets and which ones are used, are transmitted in an HTTP header.To not make this too complex Flask just assumes that if you are sendingUnicode out you want it to be UTF-8 encoded. Flask will do the encodingand setting of the appropriate headers for you.

The same is true if you are talking to databases with the help ofSQLAlchemy or a similar ORM system. Some databases have a protocol thatalready transmits Unicode and if they do not, SQLAlchemy or your other ORMshould take care of that.

The Golden Rule

So the rule of thumb: if you are not dealing with binary data, work withUnicode. What does working with Unicode in Python 2.x mean?

  • as long as you are using ASCII code points only (basically numbers,some special characters of Latin letters without umlauts or anythingfancy) you can use regular string literals ('Hello World').

  • if you need anything else than ASCII in a string you have to markthis string as Unicode string by prefixing it with a lowercase u.(like u'Hänsel und Gretel')

  • if you are using non-Unicode characters in your Python files you haveto tell Python which encoding your file uses. Again, I recommendUTF-8 for this purpose. To tell the interpreter your encoding you canput the # -- coding: utf-8 -- into the first or second line ofyour Python source file.

  • Jinja is configured to decode the template files from UTF-8. So makesure to tell your editor to save the file as UTF-8 there as well.

Encoding and Decoding Yourself

If you are talking with a filesystem or something that is not really basedon Unicode you will have to ensure that you decode properly when workingwith Unicode interface. So for example if you want to load a file on thefilesystem and embed it into a Jinja2 template you will have to decode itfrom the encoding of that file. Here the old problem that text files donot specify their encoding comes into play. So do yourself a favour andlimit yourself to UTF-8 for text files as well.

Anyways. To load such a file with Unicode you can use the built-instr.decode() method:

  1. def read_file(filename, charset='utf-8'):
  2.     with open(filename, 'r') as f:
  3.         return f.read().decode(charset)

To go from Unicode into a specific charset such as UTF-8 you can use theunicode.encode() method:

  1. def write_file(filename, contents, charset='utf-8'):
  2.     with open(filename, 'w') as f:
  3.         f.write(contents.encode(charset))

Configuring Editors

Most editors save as UTF-8 by default nowadays but in case your editor isnot configured to do this you have to change it. Here some common ways toset your editor to store as UTF-8:

  • Vim: put set enc=utf-8 to your .vimrc file.

  • Emacs: either use an encoding cookie or put this into your .emacsfile:

  1. (prefer-coding-system 'utf-8)
  2. (setq default-buffer-file-coding-system 'utf-8)

  • Notepad++:

    • Go to Settings -> Preferences …

    • Select the “New Document/Default Directory” tab

    • Select “UTF-8 without BOM” as encoding

It is also recommended to use the Unix newline format, you can selectit in the same panel but this is not a requirement.