From: eLinux.org

KDB

Contents

Introduction and basic resources

Here is some information about KDB - the in-kernel debugger for the
Linux kernel.

The KDB and KGDB official wiki:
https://kgdb.wiki.kernel.org/ (this
only has 2 pages?)

Jason Wessel is the current KDB maintainer. Here is a presentation from
him at LinuxCon 2010 (August 2010):
http://kernel.org/pub/linux/kernel/people/jwessel/dbg_webinar/State_Of_kernel_debugging_LinuxCon2010.pdf

Here are some videos showing use of KDB and KGDB:

Documentation, up-to-date as of 2010, for KDB and KGDB is at:
http://kernel.org/pub/linux/kernel/people/jwessel/kdb/

Older Information

See
http://www.ibm.com/developerworks/linux/library/l-kdbug/
for a tutorial for the 2.4.20 kernel (from June 2003)

Here’s an article from 2002 on KDB vs. KGDB:
http://kerneltrap.org/node/112 It has
a good discussion excerpt between Andrew Morton and Keith Owens about
the relative merits of KDB versus KGDB.

General Information

Kernel Versions supported

kgdb was added to the mainline Linux kernel in version 2.6.26.

kdb support was added to the mainline Linux kernel in version 2.6.35.

Before those versions, kgdb and kdb were available as patches which
could be applied to the Linux kernel source.

Kernel configuration

The following descriptions are for a 2.6.35 kernel, using KDB over a
serial line between host and target:

All these options on are the “Kernel Hacking” menu.

In order to support KDB, “KGDB” support must be turned on first (even if
you aren’t using kgdb/gdb)

Config variables

  • CONFIG_DEBUG_KERNEL=Y - includes debug information in the kernel
    compilation - required for basic kernel debugging support
  • CONFIG_KGDB=Y - turn on basic kernel debug agent support
  • CONFIG_KGDB_SERIAL_CONSOLE=Y - to share a serial console with
    kgdb.
    • Sysrq-g must be used to break in initially.
    • Selecting this will automatically set:
      • CONFIG_CONSOLE_POLL=Y
      • CONFIG_MAGIC_SYSRQ=Y - turn on MAGIC-SYSRQ key support
  • CONFIG_KGDB_KDB=Y - actually turn on the KDB debugger feature
  • CONFIG_DEBUG_RODATA=N - you must disable this on x86 in order to
    set breakpoints on code or data

Optional other configuration settings:

  • CONFIG_FRAME_POINTER=Y - this allows for better backtrace support
    in KDB
  • CONFIG_KALLSYMS=Y - this adds symbolic information to the kernel,
    useful to see symbols instead of addresses
  • CONFIG_KDB_KEYBOARD - use KDB with an attached keyboard (not for
    use with serial console)
  • CONFIG_KGDB_TESTS - used to turn on kgdb internal self-tests - see
    the config help for this for more information

Steps

To turn on KDB over serial console, do the following:

  • ‘make menuconfig’
    • go to “Kernel Hacking” sub-menu
    • turn on “KGDB: kernel debugger”, and choose “\“ to go
      to sub-menu
    • verify that “KGDB: use kgdb over the serial console” is set
    • set “KGDB_KDB: include kdb frontend for kgdb”
  • save and exit

Results:

  1. []$ diffconfig
  2. KGDB n -> y
  3. +CONSOLE_POLL y
  4. +KDB_KEYBOARD n
  5. +KGDB_KDB y
  6. +KGDB_SERIAL_CONSOLE y
  7. +KGDB_TESTS n

Enabling kdb

At runtime

Once the kernel is compiled with kdb support and is running on your
target board, you need to enable it. This can be done on a running
system, binding the kdb/kgdb feature to a serial port, by writing a
value into the sys filesystem.

If your machine starts a serial console on ttyS0, you can bind kdb/kgdb
to this serial console by writing the string “ttyS0” to
/sys/module/kgdboc/parameters/kgdboc. The kernel will respond with a
message indicating that that driver is registered.

  1. $ echo ttyS0 >/sys/module/kgdboc/parameters/kgdboc
  2. kgdb: Registered I/O driver kgdboc.

At boot time (via kernel command line)

You can enable kdb support in your kernel at boot time by using the
‘kgdboc’ option on the kernel command line. Normally, you specify the
tty device name, followed by the serial port speed.

  1. kgdboc=ttyS0,115200

Invoking kdb

Once the kernel is running, and the kgdb/kdb is bound to the serial
console, you can invoke the debugger in numerous ways.

First, you can enter the debugger using a Magic
SysRq command.

kdb will also be entered automatically if the kernel panics.

Finally, you can set a breakpoint (either hardware or software), such
that the debugger is invoked when the breakpoint condition is met. For a
code breakpoint, this means when the instruction is executed at the
breakpoint location, and for a data breakpoint, when the particular
access is made at the breakpoint address.

Invoking with Magic SysRq ‘g’

To invoke the debugger using the Magic SysRq command, you use the ‘g’
command, which can be issued any of the ways supported by the Magic
SysRq feature. This can be done by 1) typing the key sequence on a
connected keyboard, 2) echoing a value to /proc/sysrq-trigger, or 3)
sending a break key sequence through the serial console

Examples:

  • sysrq trigger from local shell, via procfs: ‘echo g
    >/proc/sysrq-trigger’
  • sysrq trigger via serial console break sequence:
    • in minicom, type ‘ctrl-a f g’ (quickly)
    • in telnet, through a server that supports sending a break: type

Invocation by panic

When a kernel panic occurs, then something has gone seriously wrong and
the kernel automatically enters kdb. From here you can look at memory,
do a traceback, examine registers, and do other operations to find out
more about the state of the system and debug the problem.

Invocation by breakpoint

To enter kdb using a breakpoint, first invoke kdb using the Magic SysRq
key (see above), then set a breakpoint. Then type ‘go’ to continue
execution. When the breakpoint is hit, the debugger shell will appear.

In the example that follows, items in italics are commands typed by a
user. Items following a ‘\$’ are commands entered at a shell command
(normal Linux user-space runtime), and items following ‘kgdb>‘ are
commands entered at the kdb interactive shell.

  1. $ echo g >/proc/sysrq-trigger
  2. SysRq : DEBUG
  4. Entering kdb (current=0xdfdff040, pid 71) due to Keyboard Entry
  5. kdb> bp sys_sync+4
  6. Instruction(i) BP #0 at 0xc00c9f00 (sys_sync+0x4)
  7.    is enabled  addr at 00000000c00c9f00, hardtype=0 installed=0
  9. kdb> go
  10. $ sync
  12. Entering kdb (current=0xdfdaa360, pid 72) due to Breakpoint @ 0xc00c9f00
  13. kdb> bt
  14. Stack traceback for pid 72
  15. 0xdfdaa360       72       71  1    0   R  0xdfdaa560 *sync
  16. [<c0028cb4>] (unwind_backtrace+0x0/0xe4) from [<c0026d50>] (show_stack+0x10/0x14)
  17. [<c0026d50>] (show_stack+0x10/0x14) from [<c0079e78>] (kdb_show_stack+0x58/0x80)
  18. [<c0079e78>] (kdb_show_stack+0x58/0x80) from [<c0079f1c>] (kdb_bt1.clone.0+0x7c/0xcc)
  19. [<c0079f1c>] (kdb_bt1.clone.0+0x7c/0xcc) from [<c007a240>] (kdb_bt+0x2d4/0x338)
  20. [<c007a240>] (kdb_bt+0x2d4/0x338) from [<c0078328>] (kdb_parse+0x4d4/0x5f8)
  21. [<c0078328>] (kdb_parse+0x4d4/0x5f8) from [<c0078a8c>] (kdb_main_loop+0x448/0x6b0)
  22. [<c0078a8c>] (kdb_main_loop+0x448/0x6b0) from [<c007acb4>] (kdb_stub+0x210/0x398)
  23. [<c007acb4>] (kdb_stub+0x210/0x398) from [<c0073280>] (kgdb_handle_exception+0x384/0x574)
  24. [<c0073280>] (kgdb_handle_exception+0x384/0x574) from [<c0028518>] (kgdb_brk_fn+0x18/0x20)
  25. [<c0028518>] (kgdb_brk_fn+0x18/0x20) from [<c0022198>] (do_undefinstr+0x10c/0x1a8)
  26. [<c0022198>] (do_undefinstr+0x10c/0x1a8) from [<c0022b84>] (__und_svc+0x44/0x60)
  27. Exception stack(0xdfe09f58 to 0xdfe09fa0)
  28. 9f40:                                                       00000000 bec93e74
  29. 9f60: 000437ac 00034738 00000001 bec93e74 00000049 00000024 c00230e8 dfe08000
  30. 9f80: 00000000 bec93e54 00000000 dfe09fa0 c0022f40 c00c9f00 80000013 ffffffff
  31. [<c0022b84>] (__und_svc+0x44/0x60) from [<c00c9f00>] (sys_sync+0x4/0x28)
  32. [<c00c9f00>] (sys_sync+0x4/0x28) from [<c0022f40>] (ret_fast_syscall+0x0/0x30)
  33. kdb>

This example shows an invocation of kdb, followed by setting a
breakpoint, then resuming execution with ‘go’. Then, at the Linux
user-space shell, the ‘sync’ command is run to cause the breakpoint to
occur. When kdb is entered due to the breakpoint, then ‘bt’ is run to
get a backtrace from the stack of the current process.

Using gdb to see the kernel source listing

You can use the addresses printed out in kdb, with a host-side gdb
session, to see the source code or assembly instructions around a
particular address.

The target address can come from a backtrace or register dump (e.g.
instruction pointer).

To load the source for a kernel, start gdb (or the appropriate
arch-specific gdb) with the vmlinux that matches the image running on
target. The kernel should have been compiled with debug symbols
(CONFIG_DEBUG_KERNEL=y). gdb will start, and load the symbol
information for the kernel.

Use the following commands to see various bits of information:

  • source file and line number for an instruction address
    • info line \0x\*
  • source lines around an instruction address
    • list \0x\*
  • assembly instructions at an address
    • disas 0x\, or
    • x/20i 0x\

KDB environment variables

Here are some environment variables supported by kdb (in 2.6.35):

  • LINES - set the number of lines for paging output from KDB (default
    24)
  • BTAPROMPT - prompt after each process in bta command (default 0)
  • LOGGING - if 1, echo all KDB output to the kernel log buffer (printk
    log) (default 0)
  • PROMPT - string to use for kdb interactive shell prompt (default
    ‘[%d]kdb>‘ on SMP or ‘kdb>‘ on UP)
  • MOREPROMPT - string to use for kdb paged output prompt (default
    ‘[%d]more>‘ on SMP or ‘more>‘ on UP)
  • RADIX - base used to print out (default 16)
  • MDCOUNT - number of lines of data for md command if not specified
    (default 8)
  • BYTESPERWORD - number of bytes per word for md command (default 4)
  • DTABCOUNT - number of symbols to display for tab completion (default
    30)
  • NOSECT - avoid displaying section information with md commands
    (default 1)
  • PS - specify a list of task states (one letter per state) for
    filtering the ‘ps’ command (default ‘DRSTCZEU’)
    • letters can be used from the list: DRSTCZEUIMA, each letter
      corresponding to a task state (see the PS man page for task
      states)
      • D = uninterruptible sleep
      • R = running
      • S = interruptible sleep
      • T = stopped
      • C = traced
      • Z = exited and zombied
      • E = exited and dead
      • U = unrunnable
      • I = idle
      • M = daemon
      • A = all

Unused args (in 2.6.35):

  • BTARGS - number of arguments to show in bt command (default 9)
    • is used for argcount for internal routine kdb_bt1, but argcount
      is never used in that routine
  • BTSYMARG - ??? [set in kdb_cmds ‘dumpall’ and ‘dumpcpu’, but not
    used in code)
    • appears to be an integer

Hints and tips

Command line completion

kdb supports command line completion using the tab key. While typing at
the interactive shell, you can press the tab key to get a list of
symbols to use as part of the command. If you type the first few letters
of a symbol name, then press the tab key, the shell will complete the
name for you. If more than one symbol matches what you have typed, then
a list is shown of matching symbols. The number of symbols shown is
controlled by the DTABCOUNT variable.

Defining a set of commands

You can define a set of commands to be run as a group (in sequence),
using the ‘defcmd’ command. This group essentially becomes a new command
which you can run, using the indicated name.

The second two strings after the new command name are the Usage string
and the Description string. These appear in the online help when the
‘help’ or ‘?’ commands are run.

Here is an example of the definition of a new command:

  1. defcmd dumpcommon "" "Common kdb debugging"
  2.   set BTAPROMPT 0
  3.   set LINES 10000
  4.   -summary
  5.   -cpu
  6.   -ps
  7.   -dmesg 600
  8.   -bt
  9. endefcmd

A leading dash indicates to ignore any errors processing a command.

Executing commands on kdb initialization

In the kernel source, you can place commands in the file ‘kdb_cmds’.
These commands will be executed by kdb on it’s initialization. By
default, a few command sets are declared: ‘dumpcommon’, ‘dumpall’, and
‘dumpcpu’.

Piping command results through ‘grep’

kdb includes a grep-like capability, which provides the ability to
filter command results using a pattern. To use this, type the regular
command, and at the end of the line add ‘| grep \‘, where
pattern is the pattern to match.

To see online help for this feature, type ‘grephelp’ at the kdb shell
prompt.

The pattern may include a very limited set of metacharacters:

  1.   pattern or ^pattern or pattern$ or ^pattern$

And if there are spaces in the pattern, you may quote it:

  1.   "pat tern" or "^pat tern" or "pat tern$" or "^pat tern$"

Command history

You can access the command history by typing up-arrow and down-arrow,
and select a previous command to run.

If you hit \ without typing a command, sometimes (when it makes
sense) the last command run will be run again. For ‘md’ commands, the
next command will continue at the address where the last command left
off.

Feature extension notes (by Tim)

This section just has some random notes on Tim Bird’s investigation of
KDB (for kernel version 2.6.35)

Regular users of KDB can ignore this…

  • Can break on __do_user_fault, to break into kdb on user process
    SIGSEGV (on ARM, anyway)

  • See CONFIG_DEBUG_USER (on ARM) for extra information shown and
    code paths used, on user-space faults

  • I can’t set a breakpoint from kdb_cmds (on kdb initialization)

    • I get a hang (likely a panic before the console starts) on any
      breakpoint set on kdb initialization
      • how to debug: 1) printk and logbuf visibility over reset? 2)
        use of qemu??

internals

internal functions to retrieve environment variables:

  • kdbgetintenv()
  • kdbgetenv()
  • kdbgetulenv() - get unsigned long env variable

Category: