22. gdb Support

If you experience low-level problems such as crashes or deadlocks (e.g. when tinkering with parts of CPython which are written in C), it can be convenient to use a low-level debugger such as gdb in order to diagnose and fix the issue. By default, however, gdb (or any of its front-ends) doesn’t know about high-level information specific to the CPython interpreter, such as which Python function is currently executing, or what type or value has a given Python object represented by a standard PyObject * pointer. We hereafter present two ways to overcome this limitation.

22.1. gdb 7 and later

In gdb 7, support for extending gdb with Python was added. When CPython is built you will notice a python-gdb.py file in the root directory of your checkout. Read the module docstring for details on how to use the file to enhance gdb for easier debugging of a CPython process.

To activate support, you must add the directory containing python-gdb.py to GDB’s “auto-load-safe-path”. Put this in your ~/.gdbinit file:

  1. add-auto-load-safe-path /path/to/checkout

You can also add multiple paths, separated by :.

This is what a backtrace looks like (truncated) when this extension is enabled:

  1. #0  0x000000000041a6b1 in PyObject_Malloc (nbytes=Cannot access memory at address 0x7fffff7fefe8
  2. ) at Objects/obmalloc.c:748
  3. #1  0x000000000041b7c0 in _PyObject_DebugMallocApi (id=111 'o', nbytes=24) at Objects/obmalloc.c:1445
  4. #2  0x000000000041b717 in _PyObject_DebugMalloc (nbytes=24) at Objects/obmalloc.c:1412
  5. #3  0x000000000044060a in _PyUnicode_New (length=11) at Objects/unicodeobject.c:346
  6. #4  0x00000000004466aa in PyUnicodeUCS2_DecodeUTF8Stateful (s=0x5c2b8d "__lltrace__", size=11, errors=0x0, consumed=
  7.     0x0) at Objects/unicodeobject.c:2531
  8. #5  0x0000000000446647 in PyUnicodeUCS2_DecodeUTF8 (s=0x5c2b8d "__lltrace__", size=11, errors=0x0)
  9.     at Objects/unicodeobject.c:2495
  10. #6  0x0000000000440d1b in PyUnicodeUCS2_FromStringAndSize (u=0x5c2b8d "__lltrace__", size=11)
  11.     at Objects/unicodeobject.c:551
  12. #7  0x0000000000440d94 in PyUnicodeUCS2_FromString (u=0x5c2b8d "__lltrace__") at Objects/unicodeobject.c:569
  13. #8  0x0000000000584abd in PyDict_GetItemString (v=
  14.     {'Yuck': <type at remote 0xad4730>, '__builtins__': <module at remote 0x7ffff7fd5ee8>, '__file__': 'Lib/test/crashers/nasty_eq_vs_dict.py', '__package__': None, 'y': <Yuck(i=0) at remote 0xaacd80>, 'dict': {0: 0, 1: 1, 2: 2, 3: 3}, '__cached__': None, '__name__': '__main__', 'z': <Yuck(i=0) at remote 0xaace60>, '__doc__': None}, key=
  15.     0x5c2b8d "__lltrace__") at Objects/dictobject.c:2171

(Notice how the dictionary argument to PyDict_GetItemString is displayed as its repr(), rather than an opaque PyObject * pointer.)

The extension works by supplying a custom printing routine for values of type PyObject *. If you need to access lower-level details of an object, then cast the value to a pointer of the appropriate type. For example:

  1. (gdb) p globals
  2. $1 = {'__builtins__': <module at remote 0x7ffff7fb1868>, '__name__':
  3. '__main__', 'ctypes': <module at remote 0x7ffff7f14360>, '__doc__': None,
  4. '__package__': None}
  6. (gdb) p *(PyDictObject*)globals
  7. $2 = {ob_refcnt = 3, ob_type = 0x3dbdf85820, ma_fill = 5, ma_used = 5,
  8. ma_mask = 7, ma_table = 0x63d0f8, ma_lookup = 0x3dbdc7ea70
  9. <lookdict_string>, ma_smalltable = {{me_hash = 7065186196740147912,
  10. me_key = '__builtins__', me_value = <module at remote 0x7ffff7fb1868>},
  11. {me_hash = -368181376027291943, me_key = '__name__',
  12. me_value ='__main__'}, {me_hash = 0, me_key = 0x0, me_value = 0x0},
  13. {me_hash = 0, me_key = 0x0, me_value = 0x0},
  14. {me_hash = -9177857982131165996, me_key = 'ctypes',
  15. me_value = <module at remote 0x7ffff7f14360>},
  16. {me_hash = -8518757509529533123, me_key = '__doc__', me_value = None},
  17. {me_hash = 0, me_key = 0x0, me_value = 0x0}, {
  18.   me_hash = 6614918939584953775, me_key = '__package__', me_value = None}}}

The pretty-printers try to closely match the repr() implementation of the underlying implementation of Python, and thus vary somewhat between Python 2 and Python 3.

An area that can be confusing is that the custom printer for some types look a lot like gdb’s built-in printer for standard types. For example, the pretty-printer for a Python 3 int gives a repr() that is not distinguishable from a printing of a regular machine-level integer:

  1. (gdb) p some_machine_integer
  2. $3 = 42
  4. (gdb) p some_python_integer
  5. $4 = 42
  7. (gdb) p *(PyLongObject*)some_python_integer
  8. $5 = {ob_base = {ob_base = {ob_refcnt = 8, ob_type = 0x3dad39f5e0}, ob_size = 1},
  9. ob_digit = {42}}

A similar confusion can arise with the str type, where the output looks a lot like gdb’s built-in printer for char *:

  1. (gdb) p ptr_to_python_str
  2. $6 = '__builtins__'

The pretty-printer for str instances defaults to using single-quotes (as does Python’s repr for strings) whereas the standard printer for char * values uses double-quotes and contains a hexadecimal address:

  1. (gdb) p ptr_to_char_star
  2. $7 = 0x6d72c0 "hello world"

Here’s how to see the implementation details of a str instance (for Python 3, where a str is a PyUnicodeObject *):

  1. (gdb) p *(PyUnicodeObject*)$6
  2. $8 = {ob_base = {ob_refcnt = 33, ob_type = 0x3dad3a95a0}, length = 12,
  3. str = 0x7ffff2128500, hash = 7065186196740147912, state = 1, defenc = 0x0}

As well as adding pretty-printing support for PyObject *, the extension adds a number of commands to gdb:

py-list

List the Python source code (if any) for the current frame in the selected thread. The current line is marked with a “>”:

  1. (gdb) py-list
  2.  901        if options.profile:
  3.  902            options.profile = False
  4.  903            profile_me()
  5.  904            return
  6.  905
  7. >906        u = UI()
  8.  907        if not u.quit:
  9.  908            try:
  10.  909                gtk.main()
  11.  910            except KeyboardInterrupt:
  12.  911                # properly quit on a keyboard interrupt...

Use py-list START to list at a different line number within the python source, and py-list START,END to list a specific range of lines within the python source.

py-up and py-down

The py-up and py-down commands are analogous to gdb’s regular up and down commands, but try to move at the level of CPython frames, rather than C frames.

gdb is not always able to read the relevant frame information, depending on the optimization level with which CPython was compiled. Internally, the commands look for C frames that are executing PyEval_EvalFrameEx (which implements the core bytecode interpreter loop within CPython) and look up the value of the related PyFrameObject *.

They emit the frame number (at the C level) within the thread.

For example:

  1. (gdb) py-up
  2. #37 Frame 0x9420b04, for file /usr/lib/python2.6/site-packages/
  3. gnome_sudoku/main.py, line 906, in start_game ()
  4.     u = UI()
  5. (gdb) py-up
  6. #40 Frame 0x948e82c, for file /usr/lib/python2.6/site-packages/
  7. gnome_sudoku/gnome_sudoku.py, line 22, in start_game(main=<module at remote 0xb771b7f4>)
  8.     main.start_game()
  9. (gdb) py-up
  10. Unable to find an older python frame

so we’re at the top of the python stack. Going back down:

  1. (gdb) py-down
  2. #37 Frame 0x9420b04, for file /usr/lib/python2.6/site-packages/gnome_sudoku/main.py, line 906, in start_game ()
  3.     u = UI()
  4. (gdb) py-down
  5. #34 (unable to read python frame information)
  6. (gdb) py-down
  7. #23 (unable to read python frame information)
  8. (gdb) py-down
  9. #19 (unable to read python frame information)
  10. (gdb) py-down
  11. #14 Frame 0x99262ac, for file /usr/lib/python2.6/site-packages/gnome_sudoku/game_selector.py, line 201, in run_swallowed_dialog (self=<NewOrSavedGameSelector(new_game_model=<gtk.ListStore at remote 0x98fab44>, puzzle=None, saved_games=[{'gsd.auto_fills': 0, 'tracking': {}, 'trackers': {}, 'notes': [], 'saved_at': 1270084485, 'game': '7 8 0 0 0 0 0 5 6 0 0 9 0 8 0 1 0 0 0 4 6 0 0 0 0 7 0 6 5 0 0 0 4 7 9 2 0 0 0 9 0 1 0 0 0 3 9 7 6 0 0 0 1 8 0 6 0 0 0 0 2 8 0 0 0 5 0 4 0 6 0 0 2 1 0 0 0 0 0 4 5\n7 8 0 0 0 0 0 5 6 0 0 9 0 8 0 1 0 0 0 4 6 0 0 0 0 7 0 6 5 1 8 3 4 7 9 2 0 0 0 9 0 1 0 0 0 3 9 7 6 0 0 0 1 8 0 6 0 0 0 0 2 8 0 0 0 5 0 4 0 6 0 0 2 1 0 0 0 0 0 4 5', 'gsd.impossible_hints': 0, 'timer.__absolute_start_time__': <float at remote 0x984b474>, 'gsd.hints': 0, 'timer.active_time': <float at remote 0x984b494>, 'timer.total_time': <float at remote 0x984b464>}], dialog=<gtk.Dialog at remote 0x98faaa4>, saved_game_model=<gtk.ListStore at remote 0x98fad24>, sudoku_maker=<SudokuMaker(terminated=False, played=[], batch_siz...(truncated)
  12.             swallower.run_dialog(self.dialog)
  13. (gdb) py-down
  14. #11 Frame 0x9aead74, for file /usr/lib/python2.6/site-packages/gnome_sudoku/dialog_swallower.py, line 48, in run_dialog (self=<SwappableArea(running=<gtk.Dialog at remote 0x98faaa4>, main_page=0) at remote 0x98fa6e4>, d=<gtk.Dialog at remote 0x98faaa4>)
  15.             gtk.main()
  16. (gdb) py-down
  17. #8 (unable to read python frame information)
  18. (gdb) py-down
  19. Unable to find a newer python frame

and we’re at the bottom of the python stack.

py-bt

The py-bt command attempts to display a Python-level backtrace of the current thread.

For example:

  1. (gdb) py-bt
  2. #8 (unable to read python frame information)
  3. #11 Frame 0x9aead74, for file /usr/lib/python2.6/site-packages/gnome_sudoku/dialog_swallower.py, line 48, in run_dialog (self=<SwappableArea(running=<gtk.Dialog at remote 0x98faaa4>, main_page=0) at remote 0x98fa6e4>, d=<gtk.Dialog at remote 0x98faaa4>)
  4.             gtk.main()
  5. #14 Frame 0x99262ac, for file /usr/lib/python2.6/site-packages/gnome_sudoku/game_selector.py, line 201, in run_swallowed_dialog (self=<NewOrSavedGameSelector(new_game_model=<gtk.ListStore at remote 0x98fab44>, puzzle=None, saved_games=[{'gsd.auto_fills': 0, 'tracking': {}, 'trackers': {}, 'notes': [], 'saved_at': 1270084485, 'game': '7 8 0 0 0 0 0 5 6 0 0 9 0 8 0 1 0 0 0 4 6 0 0 0 0 7 0 6 5 0 0 0 4 7 9 2 0 0 0 9 0 1 0 0 0 3 9 7 6 0 0 0 1 8 0 6 0 0 0 0 2 8 0 0 0 5 0 4 0 6 0 0 2 1 0 0 0 0 0 4 5\n7 8 0 0 0 0 0 5 6 0 0 9 0 8 0 1 0 0 0 4 6 0 0 0 0 7 0 6 5 1 8 3 4 7 9 2 0 0 0 9 0 1 0 0 0 3 9 7 6 0 0 0 1 8 0 6 0 0 0 0 2 8 0 0 0 5 0 4 0 6 0 0 2 1 0 0 0 0 0 4 5', 'gsd.impossible_hints': 0, 'timer.__absolute_start_time__': <float at remote 0x984b474>, 'gsd.hints': 0, 'timer.active_time': <float at remote 0x984b494>, 'timer.total_time': <float at remote 0x984b464>}], dialog=<gtk.Dialog at remote 0x98faaa4>, saved_game_model=<gtk.ListStore at remote 0x98fad24>, sudoku_maker=<SudokuMaker(terminated=False, played=[], batch_siz...(truncated)
  6.             swallower.run_dialog(self.dialog)
  7. #19 (unable to read python frame information)
  8. #23 (unable to read python frame information)
  9. #34 (unable to read python frame information)
  10. #37 Frame 0x9420b04, for file /usr/lib/python2.6/site-packages/gnome_sudoku/main.py, line 906, in start_game ()
  11.     u = UI()
  12. #40 Frame 0x948e82c, for file /usr/lib/python2.6/site-packages/gnome_sudoku/gnome_sudoku.py, line 22, in start_game (main=<module at remote 0xb771b7f4>)
  13.     main.start_game()

The frame numbers correspond to those displayed by gdb’s standard backtrace command.

py-print

The py-print command looks up a Python name and tries to print it. It looks in locals within the current thread, then globals, then finally builtins:

  1. (gdb) py-print self
  2. local 'self' = <SwappableArea(running=<gtk.Dialog at remote 0x98faaa4>,
  3. main_page=0) at remote 0x98fa6e4>
  4. (gdb) py-print __name__
  5. global '__name__' = 'gnome_sudoku.dialog_swallower'
  6. (gdb) py-print len
  7. builtin 'len' = <built-in function len>
  8. (gdb) py-print scarlet_pimpernel
  9. 'scarlet_pimpernel' not found

py-locals

The py-locals command looks up all Python locals within the current Python frame in the selected thread, and prints their representations:

  1. (gdb) py-locals
  2. self = <SwappableArea(running=<gtk.Dialog at remote 0x98faaa4>,
  3. main_page=0) at remote 0x98fa6e4>
  4. d = <gtk.Dialog at remote 0x98faaa4>

You can of course use other gdb commands. For example, the frame command takes you directly to a particular frame within the selected thread. We can use it to go a specific frame shown by py-bt like this:

  1. (gdb) py-bt
  2. (output snipped)
  3. #68 Frame 0xaa4560, for file Lib/test/regrtest.py, line 1548, in <module> ()
  4.         main()
  5. (gdb) frame 68
  6. #68 0x00000000004cd1e6 in PyEval_EvalFrameEx (f=Frame 0xaa4560, for file Lib/test/regrtest.py, line 1548, in <module> (), throwflag=0) at Python/ceval.c:2665
  7. 2665                            x = call_function(&sp, oparg);
  8. (gdb) py-list
  9. 1543        # Run the tests in a context manager that temporary changes the CWD to a
  10. 1544        # temporary and writable directory. If it's not possible to create or
  11. 1545        # change the CWD, the original CWD will be used. The original CWD is
  12. 1546        # available from test_support.SAVEDCWD.
  13. 1547        with test_support.temp_cwd(TESTCWD, quiet=True):
  14. >1548            main()

The info threads command will give you a list of the threads within the process, and you can use the thread command to select a different one:

  1. (gdb) info threads
  2.   105 Thread 0x7fffefa18710 (LWP 10260)  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
  3.   104 Thread 0x7fffdf5fe710 (LWP 10259)  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
  4. * 1 Thread 0x7ffff7fe2700 (LWP 10145)  0x00000038e46d73e3 in select () at ../sysdeps/unix/syscall-template.S:82

You can use thread apply all COMMAND or (t a a COMMAND for short) to run a command on all threads. You can use this with py-bt to see what every thread is doing at the Python level:

  1. (gdb) t a a py-bt
  3. Thread 105 (Thread 0x7fffefa18710 (LWP 10260)):
  4. #5 Frame 0x7fffd00019d0, for file /home/david/coding/python-svn/Lib/threading.py, line 155, in _acquire_restore (self=<_RLock(_Verbose__verbose=False, _RLock__owner=140737354016512, _RLock__block=<thread.lock at remote 0x858770>, _RLock__count=1) at remote 0xd7ff40>, count_owner=(1, 140737213728528), count=1, owner=140737213728528)
  5.         self.__block.acquire()
  6. #8 Frame 0x7fffac001640, for file /home/david/coding/python-svn/Lib/threading.py, line 269, in wait (self=<_Condition(_Condition__lock=<_RLock(_Verbose__verbose=False, _RLock__owner=140737354016512, _RLock__block=<thread.lock at remote 0x858770>, _RLock__count=1) at remote 0xd7ff40>, acquire=<instancemethod at remote 0xd80260>, _is_owned=<instancemethod at remote 0xd80160>, _release_save=<instancemethod at remote 0xd803e0>, release=<instancemethod at remote 0xd802e0>, _acquire_restore=<instancemethod at remote 0xd7ee60>, _Verbose__verbose=False, _Condition__waiters=[]) at remote 0xd7fd10>, timeout=None, waiter=<thread.lock at remote 0x858a90>, saved_state=(1, 140737213728528))
  7.             self._acquire_restore(saved_state)
  8. #12 Frame 0x7fffb8001a10, for file /home/david/coding/python-svn/Lib/test/lock_tests.py, line 348, in f ()
  9.             cond.wait()
  10. #16 Frame 0x7fffb8001c40, for file /home/david/coding/python-svn/Lib/test/lock_tests.py, line 37, in task (tid=140737213728528)
  11.                 f()
  13. Thread 104 (Thread 0x7fffdf5fe710 (LWP 10259)):
  14. #5 Frame 0x7fffe4001580, for file /home/david/coding/python-svn/Lib/threading.py, line 155, in _acquire_restore (self=<_RLock(_Verbose__verbose=False, _RLock__owner=140737354016512, _RLock__block=<thread.lock at remote 0x858770>, _RLock__count=1) at remote 0xd7ff40>, count_owner=(1, 140736940992272), count=1, owner=140736940992272)
  15.         self.__block.acquire()
  16. #8 Frame 0x7fffc8002090, for file /home/david/coding/python-svn/Lib/threading.py, line 269, in wait (self=<_Condition(_Condition__lock=<_RLock(_Verbose__verbose=False, _RLock__owner=140737354016512, _RLock__block=<thread.lock at remote 0x858770>, _RLock__count=1) at remote 0xd7ff40>, acquire=<instancemethod at remote 0xd80260>, _is_owned=<instancemethod at remote 0xd80160>, _release_save=<instancemethod at remote 0xd803e0>, release=<instancemethod at remote 0xd802e0>, _acquire_restore=<instancemethod at remote 0xd7ee60>, _Verbose__verbose=False, _Condition__waiters=[]) at remote 0xd7fd10>, timeout=None, waiter=<thread.lock at remote 0x858860>, saved_state=(1, 140736940992272))
  17.             self._acquire_restore(saved_state)
  18. #12 Frame 0x7fffac001c90, for file /home/david/coding/python-svn/Lib/test/lock_tests.py, line 348, in f ()
  19.             cond.wait()
  20. #16 Frame 0x7fffac0011c0, for file /home/david/coding/python-svn/Lib/test/lock_tests.py, line 37, in task (tid=140736940992272)
  21.                 f()
  23. Thread 1 (Thread 0x7ffff7fe2700 (LWP 10145)):
  24. #5 Frame 0xcb5380, for file /home/david/coding/python-svn/Lib/test/lock_tests.py, line 16, in _wait ()
  25.     time.sleep(0.01)
  26. #8 Frame 0x7fffd00024a0, for file /home/david/coding/python-svn/Lib/test/lock_tests.py, line 378, in _check_notify (self=<ConditionTests(_testMethodName='test_notify', _resultForDoCleanups=<TestResult(_original_stdout=<cStringIO.StringO at remote 0xc191e0>, skipped=[], _mirrorOutput=False, testsRun=39, buffer=False, _original_stderr=<file at remote 0x7ffff7fc6340>, _stdout_buffer=<cStringIO.StringO at remote 0xc9c7f8>, _stderr_buffer=<cStringIO.StringO at remote 0xc9c790>, _moduleSetUpFailed=False, expectedFailures=[], errors=[], _previousTestClass=<type at remote 0x928310>, unexpectedSuccesses=[], failures=[], shouldStop=False, failfast=False) at remote 0xc185a0>, _threads=(0,), _cleanups=[], _type_equality_funcs={<type at remote 0x7eba00>: <instancemethod at remote 0xd750e0>, <type at remote 0x7e7820>: <instancemethod at remote 0xd75160>, <type at remote 0x7e30e0>: <instancemethod at remote 0xd75060>, <type at remote 0x7e7d20>: <instancemethod at remote 0xd751e0>, <type at remote 0x7f19e0...(truncated)
  27.         _wait()

Note

This is only available for Python 2.7, 3.2 and higher.

22.2. gdb 6 and earlier

The file at Misc/gdbinit contains a gdb configuration file which provides extra commands when working with a CPython process. To register these commands permanently, either copy the commands to your personal gdb configuration file or symlink ~/.gdbinit to Misc/gdbinit. To use these commands from a single gdb session without registering them, type source Misc/gdbinit from your gdb session.

22.3. Updating auto-load-safe-path to allow test_gdb to run

test_gdb attempts to automatically load additional Python specific hooks into gdb in order to test them. Unfortunately, the command line options it uses to do this aren’t always supported correctly.

If test_gdb is being skipped with an “auto-loading has been declined” message, then it is necessary to identify any Python build directories as auto-load safe. One way to achieve this is to add a line like the following to ~/.gdbinit (edit the specific list of paths as appropriate):

  1. add-auto-load-safe-path ~/devel/py3k:~/devel/py32:~/devel/py27