The uWSGI Emperor – multi-app deployment

The uWSGI Emperor – multi-app deployment

If you need to deploy a big number of apps on a single server, or a group ofservers, the Emperor mode is just the ticket. It is a special uWSGI instancethat will monitor specific events and will spawn/stop/reload instances (knownas vassals, when managed by an Emperor) on demand.

By default the Emperor will scan specific directories for supported (.ini,.xml, .yml, .json, etc.) uWSGI configuration files, but it is extensible usingimperial monitor plugins. The dir:// and glob:// plugins areembedded in the core, so they need not be loaded, and are automaticallydetected. The dir:// plugin is the default.

Whenever an imperial monitor detects a new configuration file, a new uWSGI instance will be spawned with that configuration.
Whenever a configuration file is modified (its modification time changed, so touch —no-dereference may be your friend), the corresponding app will be reloaded.
Whenever a config file is removed, the corresponding app will be stopped.
If the emperor dies, all the vassals die.
If a vassal dies for any reason, the emperor will respawn it.

Multiple sources of configuration may be monitored by specifying —emperor multiple times.

Special configuration variables

Using Placeholders and Magic variables in conjunction with the Emperorwill probably save you a lot of time and make your configuration more DRY.Suppose that in /opt/apps there are only Django apps. /opt/apps/app.skel (the.skel extension is not a known configuration file type to uWSGI and will beskipped)

[uwsgi]
chdir = /opt/apps/%n
threads = 20
socket = /tmp/sockets/%n.sock
env = DJANGO_SETTINGS_MODULE=%n.settings
module = django.core.handlers.wsgi:WSGIHandler()

And then for each app create a symlink:

ln -s /opt/apps/app.skel /opt/apps/app1.ini
ln -s /opt/apps/app.skel /opt/apps/app2.ini

Finally, start the Emperor with the —emperor-nofollow option. Now you can reload each vassal separately with the command:

touch --no-dereference $INI_FILE

Passing configuration parameters to all vassals

Starting from 1.9.19 you can pass options using the —vassal-set facility

[uwsgi]
emperor = /etc/uwsgi/vassals
vassal-set = processes=8
vassal-set = enable-metrics=1

this will add —set processes=8 and —set enable-metrics=1 to each vassal

You can force the Emperor to pass options to uWSGI instances using environmentvariables too. Every environment variable of the form UWSGI_VASSAL_xxx will berewritten in the new instance as UWSGI_xxx, with the usualconfiguration implications.

For example:

UWSGI_VASSAL_SOCKET=/tmp/%n.sock uwsgi --emperor /opt/apps

will let you avoid specifying the socket option in configuration files.

Alternatively, you can use the —vassals-include option let eachvassal automatically include a complete config file:

uwsgi --emperor /opt/apps --vassals-include /etc/uwsgi/vassals-default.ini

Note that if you do this, %n (and other magic variables) in theincluded file will resolve to the name of the included file, not theoriginal vassal configuration file. If you want to set options in theincluded file using the vassal name, you’ll have to use placeholders.For example, in the vassal config, you write:

[uwsgi]
vassal_name = %n
... more options

In the vassal-defaults.ini, you write:

[uwsgi]
socket = /tmp/sockets/%(vassal_name).sock

Tyrant mode (secure multi-user hosting)

The emperor is normally run as root, setting the UID and GID in eachinstance’s config. The vassal instance then drops privileges before servingrequests. In this mode, if your users have access to their own uWSGIconfiguration files, you can’t trust them to set the correct uid andgid. You could run the emperor as unprivileged user (with uid andgid) but all of the vassals would then run under the same user, asunprivileged users are not able to promote themselves to other users. For thiscase the Tyrant mode is available – just add the emperor-tyrant option.

In Tyrant mode the Emperor will run the vassal using the UID/GID of the vassalconfiguration file (or for other Imperial Monitors, by some other method ofconfiguration). If Tyrant mode is used, the vassal configuration files musthave UID/GID > 0. An error will occur if the UID or GID is zero, or if the UIDor GID of the configuration of an already running vassal changes.

Tyrant mode for paranoid sysadmins (Linux only)

If you have built a uWSGI version with Setting POSIX Capabilities options enabled, youcan run the Emperor as unprivileged user but maintaining the minimal amount ofroot-capabilities needed to apply the tyrant mode

[uwsgi]
uid = 10000
gid = 10000
emperor = /tmp
emperor-tyrant = true
cap = setgid,setuid

Loyalty

As soon as a vassal manages a request it will became “loyal”. This status isused by the Emperor to identify bad-behaving vassals and punish them.

Throttling

Whenever two or more vassals are spawned in the same second, the Emperor willstart a throttling subsystem to avoid fork bombing. The system adds athrottle delta (specified in milliseconds via the OptionEmperorThrottleoption) whenever it happens, and waits for that duration before spawning a newvassal. Every time a new vassal spawns without triggering throttling, thecurrent throttling duration is halved.

Blacklist system

Whenever a non-loyal vassal dies, it is put in a shameful blacklist. When in ablacklist, that vassal will be throttled up to a maximum value (tunable viaOptionEmperorMaxThrottle), starting from the default throttle delta of3. Whenever a blacklisted vassal dies, its throttling value is increased bythe delta (OptionEmperorThrottle).

You can also empty the blacklist by sending the signal SIGURG to the emperorprocess. This will reset the throttle value.

Heartbeat system

Vassals can voluntarily ask the Emperor to monitor their status. Workers ofheartbeat-enabled vassals will send “heartbeat” messages to the Emperor. If theEmperor does not receive heartbeats from an instance for more than N (default30, OptionEmperorRequiredHeartbeat) seconds, that instance will beconsidered hung and thus reloaded. To enable sending of heartbeat packet in avassal, add the OptionHeartbeat option.

Important

If all of your workers are stuck handling perfectly legal requests such asslow, large file uploads, the Emperor will trigger a reload as if the workersare hung. The reload triggered is a graceful one, so you will be able to tuneyour config/timeout/mercy for sane behaviour.

Using Linux namespaces for vassals

On Linux you can tell the Emperor to run vassals in “unshared” contexts. That means you can run each vassal with a dedicated view of the filesystems, ipc, uts, networking, pids and uids.

Things you generally do with tools like lxc or its abstractions like docker are native in uWSGI.

For example if you want to run each vassals in a new namespace:

[uwsgi]
emperor = /etc/uwsgi/vassals
emperor-use-clone = fs,net,ipc,pid,uts

now each vassal will be able to modify the filesystem layout, networking, hostname and so on without damaging the main system.

A couple of helper daemons are included in the uWSGI distribution to simplify management of jailed vassals. Most notably The TunTap Router allows full user-space networking in jails, whilethe forkpty router allows allocation of pseudoterminals in jails

It is not needed to unshare all of the subsystem in your vassals, sometimes you only want to give dedicated ipc and hostname to a vassal and hide from the processes list:

[uwsgi]
emperor = /etc/uwsgi/vassals
emperor-use-clone = fs,ipc,pid,uts

a vassal could be:

[uwsgi]
; set the hostname
exec-as-root = hostname foobar
; umount /proc and remount to hide processes
; as we are in the 'fs' namespace umounting /proc does not interfere with the main one
exec-as-root = umount /proc
exec-as-root = mount -t proc none /proc
; drop privileges
uid = foobar
gid = foobar
; bind to the socket
socket = /tmp/myapp.socket
psgi = myapp.pl

The Imperial Bureau of Statistics

You can enable a statistics/status service for the Emperor by adding theOptionEmperorStats option with a TCP address. By connecting to thataddress, you’ll get a JSON-format blob of statistics.

Running non-uWSGI apps or using alternative uWSGIs as vassals

You can exec() a different binary as your vassal using theprivileged-binary-patch/unprivileged-binary-patch options. The firstone patches the binary after socket inheritance and shared socketinitialization (so you will be able to use uWSGI-defined sockets). The secondone patches the binary after privileges drop. In this way you will be able touse uWSGI’s UID/GID/chroot/namespace/jailing options. The binary is calledwith the same arguments that were passed to the vassal by the Emperor.

; i am a special vassal calling a different binary in a new linux network namespace
[uwsgi]
uid = 1000
gid = 1000
unshare = net
unprivileged-binary-patch = /usr/bin/myfunnyserver

Important

DO NOT DAEMONIZE your apps. If you do so, the Emperor will lose its connection with them.

The uWSGI arguments are passed to the new binary. If you do not like thatbehaviour (or need to pass custom arguments) add -arg to the binary patchoption, yielding:

; i am a special vassal calling a different binary in a new linux network namespace
; with custom options
[uwsgi]
uid = 1000
gid = 1000
unshare = net
unprivileged-binary-patch-arg = ps aux

or:

;nginx example
[uwsgi]
privileged-binary-patch-arg = nginx -g "daemon off;"

Integrating the Emperor with the FastRouter

The FastRouter is a proxy/load-balancer/router speaking The uwsgi Protocol. YannMalet from Lincoln Loop has released a draft about massive Emperor +Fastrouter deployment (PDF) using The uWSGI caching framework as a hostname to socketmapping storage.

Notes

At startup, the emperor chdir() to the vassal dir. All vassal instances will start from here.
If the uwsgi binary is not in your system path you can force its path with binary-path:

./uwsgi --emperor /opt/apps --binary-path /opt/uwsgi/uwsgi

Sending SIGUSR1 to the emperor will print vassal status in its log.
Stopping (SIGINT/SIGTERM/SIGQUIT) the Emperor will invokeRagnarok and kill all the vassals.
Sending SIGHUP to the Emperor will reload all vassals.
Sending SIGURG to the Emperor will remove all vassals from the blacklist
The emperor should generally not be run with —master, unless masterfeatures like advanced logging are specifically needed.
The emperor should generally be started at server boot time and left alone,not reloaded/restarted except for uWSGI upgrades; emperor reloads are a bitdrastic, reloading all vassals at once. Instead vassals should be reloadedindividually when needed, in the manner of the imperial monitor in use.

Todo

Docs-TODO: Clarify what the “chdir-on-startup” behavior does withnon-filesystem monitors.
Export more magic vars
Add support for multiple sections in xml/ini/yaml files (this will allow tohave a single config file for multiple instances)