Deploying your Webapp

I can’t speak for others, but I personally prefer programming to system administration. But the fact is that, eventually, you need to serve your app somehow, and odds are that you’ll need to be the one to set it up.

There are some promising initiatives in the Haskell web community towards making deployment easier. In the future, we may even have a service that allows you to deploy your app with a single command.

But we’re not there yet. And even if we were, such a solution will never work for everyone. This chapter covers the different options you have for deployment, and gives some general recommendations on what you should choose in different situations.

The Yesod team strongly recommends using the stack build tool as discussed in the quick start guide for Yesod development, check out the quick start if you haven’t already.

Keter

The Yesod scaffolding comes with some built-in support for the Keter deployment engine, which is also written in Haskell and uses many of the same underlying technologies like WAI and http-client. Keter works as a reverse proxy to your applications, as well as a system for starting, monitoring, and redeploying running apps. If you’d like to deploy with Keter, follow these steps:

  1. Edit the config/keter.yaml file in your scaffolded application as necessary.

  2. Set up some kind of server for hosting yours apps. I recommend trying Ubuntu on Amazon EC2.

  3. Install Keter on that machine. Please follow the instructions on the Keter website, which will be the most up to date.

  4. Run yesod keter to generate a Keter bundle, e.g., myapp.keter.

  5. Copy myapp.keter to the /opt/keter/incoming directory on your server.

If you’ve got things configured correctly, you should now be able to view your website, running in a production environment! In the future, upgrades can be handled by simply rerunning yesod keter and recopying the myapp.keter bundle to the server. Note that Keter will automatically detect the presence of the new file and reload your application.

The rest of this chapter will go into some more details about various steps, and provide some alternatives for people looking to either not use the scaffolding or not use Keter.

Compiling

The biggest advice I can give is: don’t compile on your server. It’s tempting to do so, as you have to just transfer source code around, and you avoid confusing dependency issues. However, compiling a Yesod application takes significant memory and CPU, which means:

  • While you’re recompiling your app, your existing application will suffer performance-wise.

  • You will need to get a much larger machine to handle compilation, and that capacity will likely sit idle most of the time, since Yesod applications tend to require far less CPU and memory than GHC itself.

Once you’re ready to compile, you should always make sure to stack clean before a new production build, to make sure no old files are lying around. Then, you can run stack build to get an executable, which will be located at dist/build/myapp/myapp.

Files to deploy

With a Yesod scaffolded application, there are essentially three sets of files that need to be deployed:

  1. Your executable.

  2. The config folder.

  3. The static folder.

Everything else, such as Shakespearean templates, gets compiled into the executable itself.

There is one caveat, however: the config/client_session_key.aes file. This file controls the server side encryption used for securing client side session cookies. Yesod will automatically generate a new one of these keys if none is present. In practice, this means that, if you do not include this file in deployment, all of your users will have to log in again when you redeploy. If you follow the advice above and include the config folder, this issue will be partially resolved. Another approach is to put your session key in an environment variable.

The other half of the resolution is to ensure that once you generate a config/client_session_key.aes file, you keep the same one for all future deployments. The simplest way to ensure this is to keep that file in your version control. However, if your version control is open source, this will be dangerous: anyone with access to your repository will be able to spoof login credentials!

The problem described here is essentially one of system administration, not programming. Yesod does not provide any built-in approach for securely storing client session keys. If you have an open source repository, or do not trust everyone who has access to your source code repository, it’s vital to figure out a safe storage solution for the client session key.

SSL and static files

There are two commonly used features in the Yesod world: serving your site over HTTPS, and placing your static files on a separate domain name. While both of these are good practices, when combined they can lead to problems if you’re not careful. In particular, most web browsers will not load up Javascript files from a non-HTTPS domain name if your HTML is served from an HTTPS domain name. In this situation, you’ll need to do one of two things:

  • Serve your static files over HTTPS as well.

  • Serve your static files from the same domain name as your main site.

Note that if you go for the first option (which is the better one), you’ll either need two separate SSL certificates, or a wildcard certificate.

Warp

As we have mentioned before, Yesod is built on the Web Application Interface (WAI), allowing it to run on any WAI backend. At the time of writing, the following backends are available:

  • Warp

  • FastCGI

  • SCGI

  • CGI

  • Webkit

  • Development server

The last two are not intended for production deployments. Of the remaining four, all can be used for production deployment in theory. In practice, a CGI backend will likely be horribly inefficient, since a new process must be spawned for each connection. And SCGI is not nearly as well supported by frontend web servers as Warp (via reverse proxying) or FastCGI.

So between the two remaining choices, Warp gets a very strong recommendation because:

  • It is significantly faster.

  • Like FastCGI, it can run behind a frontend server like Nginx, using reverse HTTP proxy.

  • In addition, it is a fully capable server of its own accord, and can therefore be used without any frontend server.

So that leaves one last question: should Warp run on its own, or via reverse proxy behind a frontend server? For most use cases, I recommend the latter, because:

  • Having a reverse proxy in front of your app makes it easier to deploy new versions.

  • Also, if you have a bug in your application, a reverse proxy can give slightly nicer error messages to users.

  • You can host multiple applications from a single host via virtual hosting.

  • Your reverse proxy can function as both a load balancer or SSL proxy as well, simplifying your application.

As discussed above, Keter is a great way to get started. If you have an existing web server running like Nginx, Yesod will work just fine sitting behind it instead.

Nginx Configuration

Keter configuration is trivial, since it is designed to work with Yesod applications. But if you want to instead use Nginx, how do you set it up?

In general, Nginx will listen on port 80 and your Yesod/Warp app will listen on some unprivileged port (lets say 4321). You will then need to provide a nginx.conf file, such as:

  1. daemon off; # Don't run nginx in the background, good for monitoring apps
  2. events {
  3.     worker_connections 4096;
  4. }
  6. http {
  7.     server {
  8.         listen 80; # Incoming port for Nginx
  9.         server_name www.myserver.com;
  10.         location / {
  11.             proxy_pass http://127.0.0.1:4321; # Reverse proxy to your Yesod app
  12.         }
  13.     }
  14. }

You can add as many server blocks as you like. A common addition is to ensure users always access your pages with the www prefix on the domain name, ensuring the RESTful principle of canonical URLs. (You could just as easily do the opposite and always strip the www, just make sure that your choice is reflected in both the nginx config and the approot of your site.) In this case, we would add the block:

  1. server {
  2.     listen 80;
  3.     server_name myserver.com;
  4.     rewrite ^/(.*) http://www.myserver.com/$1 permanent;
  5. }

A highly recommended optimization is to serve static files from a separate domain name, therefore bypassing the cookie transfer overhead. Assuming that our static files are stored in the static folder within our site folder, and the site folder is located at /home/michael/sites/mysite, this would look like:

  1. server {
  2.     listen 80;
  3.     server_name static.myserver.com;
  4.     root /home/michael/sites/mysite/static;
  5.     # Since yesod-static appends a content hash in the query string,
  6.     # we are free to set expiration dates far in the future without
  7.     # concerns of stale content.
  8.     expires max;
  9. }

In order for this to work, your site must properly rewrite static URLs to this alternate domain name. The scaffolded site is set up to make this fairly simple via the Settings.staticRoot function and the definition of urlRenderOverride. However, if you just want to get the benefit of nginx’s faster static file serving without dealing with separate domain names, you can instead modify your original server block like so:

  1. server {
  2.     listen 80; # Incoming port for Nginx
  3.     server_name www.myserver.com;
  4.     location / {
  5.         proxy_pass http://127.0.0.1:4321; # Reverse proxy to your Yesod app
  6.     }
  7.     location /static {
  8.         root /home/michael/sites/mysite; # Notice that we do *not* include /static
  9.         expires max;
  10.     }
  11. }

Server Process

Many people are familiar with an Apache/mod_php or Lighttpd/FastCGI kind of setup, where the web server automatically spawns the web application. With nginx, either for reverse proxying or FastCGI, this is not the case: you are responsible to run your own process. I strongly recommend a monitoring utility which will automatically restart your application in case it crashes. There are many great options out there, such as angel or daemontools.

To give a concrete example, here is an Upstart config file. The file must be placed in /etc/init/mysite.conf:

  1. description "My awesome Yesod application"
  2. start on runlevel [2345];
  3. stop on runlevel [!2345];
  4. respawn
  5. chdir /home/michael/sites/mysite
  6. exec /home/michael/sites/mysite/dist/build/mysite/mysite

Once this is in place, bringing up your application is as simple as sudo start mysite. A similar systemd configuration file placed in /etc/systemd/system/yesod-sample.service:

  1. [Service]
  2. ExecStart=/home/sibi/.local/bin/my-yesod-executable
  3. Restart=always
  4. StandardOutput=syslog
  5. StandardError=syslog
  6. SyslogIdentifier=yesod-sample
  8. [Install]
  9. WantedBy=multi-user.target

Now you can start your service with:

  1. systemctl enable yesod-sample
  2. systemctl start yesod-sample

You can also see the status of your process using systemctl status yesod-sample.

Nginx + FastCGI

Some people may prefer using FastCGI for deployment. In this case, you’ll need to add an extra tool to the mix. FastCGI works by receiving new connection from a file descriptor. The C library assumes that this file descriptor will be 0 (standard input), so you need to use the spawn-fcgi program to bind your application’s standard input to the correct socket.

It can be very convenient to use Unix named sockets for this instead of binding to a port, especially when hosting multiple applications on a single host. A possible script to load up your app could be:

  1. spawn-fcgi \
  2.     -d /home/michael/sites/mysite \
  3.     -s /tmp/mysite.socket \
  4.     -n \
  5.     -M 511 \
  6.     -u michael \
  7.     -- /home/michael/sites/mysite/dist/build/mysite-fastcgi/mysite-fastcgi

You will also need to configure your frontend server to speak to your app over FastCGI. This is relatively painless in Nginx:

  1. server {
  2.     listen 80;
  3.     server_name www.myserver.com;
  4.     location / {
  5.         fastcgi_pass unix:/tmp/mysite.socket;
  6.     }
  7. }

That should look pretty familiar from above. The only last trick is that, with Nginx, you need to manually specify all of the FastCGI variables. It is recommended to store these in a separate file (say, fastcgi.conf) and then add include fastcgi.conf; to the end of your http block. The contents of the file, to work with WAI, should be:

  1. fastcgi_param  QUERY_STRING       $query_string;
  2. fastcgi_param  REQUEST_METHOD     $request_method;
  3. fastcgi_param  CONTENT_TYPE       $content_type;
  4. fastcgi_param  CONTENT_LENGTH     $content_length;
  5. fastcgi_param  PATH_INFO          $fastcgi_script_name;
  6. fastcgi_param  SERVER_PROTOCOL    $server_protocol;
  7. fastcgi_param  GATEWAY_INTERFACE  CGI/1.1;
  8. fastcgi_param  SERVER_SOFTWARE    nginx/$nginx_version;
  9. fastcgi_param  REMOTE_ADDR        $remote_addr;
  10. fastcgi_param  SERVER_ADDR        $server_addr;
  11. fastcgi_param  SERVER_PORT        $server_port;
  12. fastcgi_param  SERVER_NAME        $server_name;

Desktop

Another nifty backend is wai-handler-webkit. This backend combines Warp and QtWebkit to create an executable that a user simply double-clicks. This can be a convenient way to provide an offline version of your application.

One of the very nice conveniences of Yesod for this is that your templates are all compiled into the executable, and thus do not need to be distributed with your application. Static files do, however.

There’s actually support for embedding your static files directly in the executable as well, see the yesod-static docs for more details.

A similar approach, without requiring the QtWebkit library, is wai-handler-launch, which launches a Warp server and then opens up the user’s default web browser. There’s a little trickery involved here: in order to know that the user is still using the site, wai-handler-launch inserts a “ping” Javascript snippet to every HTML page it serves. It wai-handler-launch doesn’t receive a ping for two minutes, it shuts down.

CGI on Apache

CGI and FastCGI work almost identically on Apache, so it should be fairly straight-forward to port this configuration. You essentially need to accomplish two goals:

  1. Get the server to serve your file as (Fast)CGI.

  2. Rewrite all requests to your site to go through the (Fast)CGI executable.

Here is a configuration file for serving a blog application, with an executable named “bloggy.cgi”, living in a subfolder named “blog” of the document root. This example was taken from an application living in the path /f5/snoyman/public/blog.

  1. Options +ExecCGI
  2. AddHandler cgi-script .cgi
  3. Options +FollowSymlinks
  5. RewriteEngine On
  6. RewriteRule ^/f5/snoyman/public/blog$ /blog/ [R=301,S=1]
  7. RewriteCond $1 !^bloggy.cgi
  8. RewriteCond $1 !^static/
  9. RewriteRule ^(.*) bloggy.cgi/$1 [L]

The first RewriteRule is to deal with subfolders. In particular, it redirects a request for /blog to /blog/. The first RewriteCond prevents directly requesting the executable, the second allows Apache to serve the static files, and the last line does the actual rewriting.

FastCGI on lighttpd

For this example, I’ve left off some of the basic FastCGI settings like mime-types. I also have a more complex file in production that prepends “www.” when absent and serves static files from a separate domain. However, this should serve to show the basics.

Here, “/home/michael/fastcgi” is the fastcgi application. The idea is to rewrite all requests to start with “/app”, and then serve everything beginning with “/app” via the FastCGI executable.

  1. server.port = 3000
  2. server.document-root = "/home/michael"
  3. server.modules = ("mod_fastcgi", "mod_rewrite")
  5. url.rewrite-once = (
  6.   "(.*)" => "/app/$1"
  7. )
  9. fastcgi.server = (
  10.     "/app" => ((
  11.         "socket" => "/tmp/test.fastcgi.socket",
  12.         "check-local" => "disable",
  13.         "bin-path" => "/home/michael/fastcgi", # full path to executable
  14.         "min-procs" => 1,
  15.         "max-procs" => 30,
  16.         "idle-timeout" => 30
  17.     ))
  18. )

CGI on lighttpd

This is basically the same as the FastCGI version, but tells lighttpd to run a file ending in “.cgi” as a CGI executable. In this case, the file lives at “/home/michael/myapp.cgi”.

  1. server.port = 3000
  2. server.document-root = "/home/michael"
  3. server.modules = ("mod_cgi", "mod_rewrite")
  5. url.rewrite-once = (
  6.     "(.*)" => "/myapp.cgi/$1"
  7. )
  9. cgi.assign = (".cgi" => "")