uWSGI Mules

Mules are worker processes living in the uWSGI stack but not reachable via socket connections, that can be used as a generic subsystem to offload tasks. You can see them as a more primitive spooler.

They can access the entire uWSGI API and can manage signals and be communicated with through a simple string-based message system.

To start a mule (you can start an unlimited number of them), use the mule option as many times as you need.

Mules have two modes,

  • Signal only mode (the default). In this mode the mules load your application as normal workers would. They can only respond to uWSGI signals.
  • Programmed mode. In this mode mules load a program separate from your application. See ProgrammedMules.

By default each mule starts in signal-only mode.

  1. uwsgi --socket :3031 --mule --mule --mule --mule
  1. <uwsgi>
  2.     <socket>:3031</socket>
  3.     <mule/>
  4.     <mule/>
  5.     <mule/>
  6.     <mule/>
  7. </uwsgi>

Basic usage

  1. import uwsgi
  2. from uwsgidecorators import timer, signal, filemon
  3.  
  4. # run a timer in the first available mule
  5. @timer(30, target='mule')
  6. def hello(signum):
  7.     print "Hi! I am responding to signal %d, running on mule %d" % (signum, uwsgi.mule_id())
  8.  
  9. # map signal 17 to mule 2
  10. @signal(17, target='mule2')
  11. def i_am_mule2(signum):
  12.     print "Greetings! I am running in mule number two."
  13.  
  14. # monitor /tmp and arouse all of the mules on modifications
  15. @filemon('/tmp', target='mules')
  16. def tmp_modified(signum):
  17.     print "/tmp has been modified. I am mule %d!" % uwsgi.mule_id()

Giving a brain to mules

As mentioned before, mules can be programmed. To give custom logic to a mule, give the mule option a path toa script (it must end in “.py”) or a “package.module:callable” value.

  1. uwsgi --socket :3031 --mule=somaro.py --mule=mypkg.myapp:run_mule --mule --mule

This will run 4 mules, 2 in signal-only mode, one running somaro.py and one running mypkg.myapp:run_mule.

  1. # somaro.py
  2. from threading import Thread
  3. import time
  4.  
  5. def loop1():
  6.     while True:
  7.         print "loop1: Waiting for messages... yawn."
  8.         message = uwsgi.mule_get_msg()
  9.         print message
  10.  
  11. def loop2():
  12.     print "Hi! I am loop2."
  13.     while True:
  14.         time.sleep(2)
  15.         print "This is a thread!"
  16.  
  17. t = Thread(target=loop2)
  18. t.daemon = True
  19. t.start()
  20.  
  21. if __name__ == '__main__':
  22.     loop1()

So as you can see from the example, you can use uwsgi.mule_get_msg() to receive messages in a programmed mule. Multiple threads in the same programmed mule can wait for messages.

If you want to block a mule to wait on an uWSGI signal instead of a message you can use uwsgi.signal_wait().

Use uwsgi.mule_msg() to send a message to a programmed mule. Mule messages can be sent from anywhere in the uWSGI stack, including but not limited to workers, the spoolers, another mule.

  1. # Send the string "ciuchino" to mule1.
  2. # If you do not specify a mule ID, the message will be processed by the first available programmed mule.
  3. uwsgi.mule_msg("ciuchino", 1)

As you can spawn an unlimited number of mules, you may need some form of synchronization – for example if you are developing a task management subsystem and do not want two mules to be able to start the same task simultaneously. You’re in luck – see Locks.

Organizing mules

Messages are eligible to be received by all programmed mules. They can be limited to specified groups of mules by creating mule farms using the farm option:

  1. uwsgi --socket :3031 --mule=feed.py --mule=feed.py --mule=wallow.py --farm=busy:1,2 --farm=idle:3

This will run three mules, where the mules 1 and 2, running feed.py, are members of the busy farm, and mule 3, running wallow.py is the sole member of the idle farm. Messages are retrieved exactly as in somaro.py above but using uwsgi.farm_get_msg() in place of uwsgi.mule_get_msg(). Use uwsgi.farm_msg() to send a message to a mule in a specified farm:

  1. # Send the string "eat stuff" to farm "busy"
  2. uwsgi.farm_msg("busy", "eat stuff")

As with mules, Locks can be used to ensure only a single mule in a farm receives a given message to that farm.