Python Tracebacker

New in version 1.3-dev.

Usually if you want to get a real-time traceback from your app you’d have to modify your code to add a hook or entry point for that as described on the TipsAndTricks page.

Starting from 1.3-dev, uWSGI includes a similar technique allowing you to get realtime traceback via a UNIX socket.

To enable the tracebacker, add the option py-tracebacker=<socket> where <socket> is the basename for the created UNIX sockets.

If you have 4 uWSGI workers and you add py-tracebacker=/tmp/tbsocket, four sockets named /tmp/tbsocket1 through /tmp/tbsocket4 will be created.

Connecting to one of them will return the current traceback of the threads running in the worker. To connect to those sockets you can use whatever application or method you like the best, but uWSGI includes a convenience option connect-and-read you can use:

  1. uwsgi --connect-and-read /tmp/tbsocket1

An example

Let’s write a silly test application called slow.py:

  1. import time
  2.  
  3. def dormi():
  4.   time.sleep(60)
  5.  
  6. def dormi2():
  7.   dormi()
  8.  
  9. def dormi3():
  10.   dormi2()
  11.  
  12. def dormi4():
  13.   dormi3()
  14.  
  15. def dormi5():
  16.   dormi4()
  17.  
  18. def application(e, start_response):
  19.   start_response('200 OK', [('Content-Type', 'text/html')])
  20.   dormi5()
  21.   return "hello"

And then run it:

  1. uwsgi --http :8080 -w slow --master --processes 2 --threads 4 --py-tracebacker /tmp/tbsocket.

Then make a bunch of requests into it:

  1. curl http://localhost:8080 &
  2. curl http://localhost:8080 &
  3. curl http://localhost:8080 &
  4. curl http://localhost:8080 &

Now, while these requests are running (they’ll take pretty much exactly a minute to complete each), you can retrieve the traceback for, let’s say, the two first workers:

  1. ./uwsgi --connect-and-read /tmp/tbsocket.1
  2. ./uwsgi --connect-and-read /tmp/tbsocket.2

The tracebacker output will be something like this:

  1. *** uWSGI Python tracebacker output ***
  2.  
  3. thread_id = uWSGIWorker1Core1 filename = ./slow.py lineno = 22 function = application line = dormi5()
  4. thread_id = uWSGIWorker1Core1 filename = ./slow.py lineno = 14 function = dormi5 line = def dormi5(): dormi4()
  5. thread_id = uWSGIWorker1Core1 filename = ./slow.py lineno = 13 function = dormi4 line = def dormi4(): dormi3()
  6. thread_id = uWSGIWorker1Core1 filename = ./slow.py lineno = 12 function = dormi3 line = def dormi3(): dormi2()
  7. thread_id = uWSGIWorker1Core1 filename = ./slow.py lineno = 11 function = dormi2 line = def dormi2(): dormi()
  8. thread_id = uWSGIWorker1Core1 filename = ./slow.py lineno = 9 function = dormi line = time.sleep(60)
  9.  
  10. thread_id = uWSGIWorker1Core3 filename = ./slow.py lineno = 22 function = application line = dormi5()
  11. thread_id = uWSGIWorker1Core3 filename = ./slow.py lineno = 14 function = dormi5 line = def dormi5(): dormi4()
  12. thread_id = uWSGIWorker1Core3 filename = ./slow.py lineno = 13 function = dormi4 line = def dormi4(): dormi3()
  13. thread_id = uWSGIWorker1Core3 filename = ./slow.py lineno = 12 function = dormi3 line = def dormi3(): dormi2()
  14. thread_id = uWSGIWorker1Core3 filename = ./slow.py lineno = 11 function = dormi2 line = def dormi2(): dormi()
  15. thread_id = uWSGIWorker1Core3 filename = ./slow.py lineno = 9 function = dormi line = time.sleep(60)
  16.  
  17. thread_id = MainThread filename = ./slow.py lineno = 22 function = application line = dormi5()
  18. thread_id = MainThread filename = ./slow.py lineno = 14 function = dormi5 line = def dormi5(): dormi4()
  19. thread_id = MainThread filename = ./slow.py lineno = 13 function = dormi4 line = def dormi4(): dormi3()
  20. thread_id = MainThread filename = ./slow.py lineno = 12 function = dormi3 line = def dormi3(): dormi2()
  21. thread_id = MainThread filename = ./slow.py lineno = 11 function = dormi2 line = def dormi2(): dormi()
  22. thread_id = MainThread filename = ./slow.py lineno = 9 function = dormi line = time.sleep(60)

Combining the tracebacker with Harakiri

If a request is killed by the harakiri feature, a traceback is automatically logged during the Harakiri phase.