使用 GDB 调试 Nginx

udo gdb -q -tui # -q 安静模式启动 GDB -tui 显示代码界面
进入 GDB 运行 attach Nginx 子进程报错如下：
issing separate debuginfos, use: debuginfo-install libgcc-4.8.5-4.el7.x86_64 zlib-1.2.7-17.el7.x86_64
直接安装即可

断点

ngx_process_events_and_timers
子进程即 worker 进程在运行后会停留在epoll_wait 处等待相应的事件发生，而这个函数调用被封装在 ngx_process_events_and_timers 中

  1. 1       breakpoint     keep y   0x000000000046c8f0 in ngx_process_events_and_timers at src/event/ngx_event.c:195
  2.         breakpoint already hit 39 times
  3. 2       breakpoint     keep y   0x000000000046c93e in ngx_process_events_and_timers at src/event/ngx_event.c:242
  4.         breakpoint already hit 36 times
  5. 3       breakpoint     keep y   0x000000000046c95a in ngx_process_events_and_timers at src/event/ngx_event.c:249
  6.         breakpoint already hit 35 times
  7. 4       breakpoint     keep y   0x000000000046ce7d in ngx_event_process_posted at src/event/ngx_event_posted.c:33
  8. 5       breakpoint     keep y   0x0000000000475c24 in ngx_epoll_process_events at src/event/modules/ngx_epoll_module.c:793
  9.         breakpoint already hit 25 times
  10. 6       breakpoint     keep y   0x0000000000475d28 in ngx_epoll_process_events at src/event/modules/ngx_epoll_module.c:926
  11. 7       breakpoint     keep y   0x0000000000476022 in ngx_epoll_process_events at src/event/modules/ngx_epoll_module.c:900
  12.         breakpoint already hit 28 times
  13. 8       breakpoint     keep y   0x000000000048cd4a in ngx_http_wait_request_handler at src/http/ngx_http_request.c:497
  14.         breakpoint already hit 6 times
  15. 9       breakpoint     keep y   0x000000000048cd6a in ngx_http_wait_request_handler at src/http/ngx_http_request.c:504
  16.         breakpoint already hit 7 times
  17. 10      breakpoint     keep y   0x000000000048c900 in ngx_http_process_request_line at src/http/ngx_http_request.c:945
  18.         breakpoint already hit 6 times
  19. 11      breakpoint     keep y   0x000000000048c912 in ngx_http_process_request_line at src/http/ngx_http_request.c:952
  20.         breakpoint already hit 6 times
  21. 12      breakpoint     keep y   0x000000000048ca95 in ngx_http_process_request_line at src/http/ngx_http_request.c:1011
  22. 13      breakpoint     keep y   0x000000000048cbce in ngx_http_process_request_line at src/http/ngx_http_request.c:1027
  23.         breakpoint already hit 6 times
  24. 14      breakpoint     keep y   0x000000000048c5ef in ngx_http_process_request_headers at src/http/ngx_http_request.c:1322
  25.         breakpoint already hit 7 times
  26. 15      breakpoint     keep y   0x000000000048c7ed in ngx_http_process_request_headers at src/http/ngx_http_request.c:1342
  27.         breakpoint already hit 2 times
  28. 16      breakpoint     keep y   0x000000000048c826 in ngx_http_process_request_headers at src/http/ngx_http_request.c:1348
  29.         breakpoint already hit 2 times
  30. 17      breakpoint     keep y   0x000000000048c231 in ngx_http_process_request at src/http/ngx_http_request.c:1916
  31.         breakpoint already hit 2 times
  32. 18      breakpoint     keep y   0x0000000000480757 in ngx_http_handler at src/http/ngx_http_core_module.c:839
  33.         breakpoint already hit 2 times
  34. 19      breakpoint     keep y   0x0000000000484ced in ngx_http_core_rewrite_phase at src/http/ngx_http_core_module.c:910
  35. 20      breakpoint     keep y   0x0000000000485e60 in ngx_http_core_content_phase at src/http/ngx_http_core_module.c:1372
  36.         breakpoint already hit 1 time
  37. 21      breakpoint     keep y   0x0000000000485eac in ngx_http_core_content_phase at src/http/ngx_http_core_module.c:1386
  38. 22      breakpoint     keep y   0x0000000000485e7d in ngx_http_core_content_phase at src/http/ngx_http_core_module.c:1379
  39.         breakpoint already hit 1 time
  40. 23      breakpoint     keep y   0x0000000000504c19 in ngx_http_lua_content_handler at ../ngx_lua-0.10.6/src/ngx_http_lua_contentby.c:222

Lua 请求处理

堆栈

  1. 0  __libc_writev (fd=3, vector=0x7ffe08245dd0, count=14) at ../sysdeps/unix/sysv/linux/writev.c:68
  2. 1  0x00000000004717e9 in ngx_writev (c=c@entry=0x7ff03d02f730, vec=vec@entry=0x7ffe08245db0) at src/os/unix/ngx_writev_chain.c:189
  3. 2  0x0000000000476a7e in ngx_linux_sendfile_chain (c=0x7ff03d02f730, in=0x7ff03d001418, limit=2147479551) at src/os/unix/ngx_linux_sendfile_chain.c:215
  4. 3  0x00000000004a7d15 in ngx_http_write_filter (r=0x7ff03cfffcf0, in=0x7ff03d001b78) at src/http/ngx_http_write_filter_module.c:254
  5. 4  0x00000000004a909d in ngx_http_chunked_body_filter (r=0x7ff03cfffcf0, in=<optimized out>) at src/http/modules/ngx_http_chunked_filter_module.c:224
  6. 5  0x00000000004ac7dc in ngx_http_gzip_body_filter (r=0x7ff03cfffcf0, in=0x7ffe082466d0) at src/http/modules/ngx_http_gzip_filter_module.c:326
  7. 6  0x00000000004afd95 in ngx_http_ssi_body_filter (r=0x7ff03cfffcf0, in=<optimized out>) at src/http/modules/ngx_http_ssi_filter_module.c:411
  8. 7  0x00000000004b2cf0 in ngx_http_charset_body_filter (r=0x7ff03cfffcf0, in=0x7ffe082466d0) at src/http/modules/ngx_http_charset_filter_module.c:647
  9. 8  0x0000000000506c7c in ngx_http_lua_capture_body_filter (r=0x7ff03cfffcf0, in=0x7ffe082466d0) at ../ngx_lua-0.10.6/src/ngx_http_lua_capturefilter.c:133
  10. 9  0x0000000000453395 in ngx_output_chain (ctx=ctx@entry=0x7ff03d001428, in=in@entry=0x7ffe082466d0) at src/core/ngx_output_chain.c:74
  11. 10 0x00000000004b4f95 in ngx_http_copy_filter (r=0x7ff03cfffcf0, in=0x7ffe082466d0) at src/http/ngx_http_copy_filter_module.c:152
  12. 11 0x0000000000485a37 in ngx_http_output_filter (r=r@entry=0x7ff03cfffcf0, in=in@entry=0x7ffe082466d0) at src/http/ngx_http_core_module.c:1979
  13. 12 0x0000000000489e33 in ngx_http_send_special (r=r@entry=0x7ff03cfffcf0, flags=flags@entry=1) at src/http/ngx_http_request.c:3358
  14. 13 0x00000000004ffc38 in ngx_http_lua_send_special (flags=1, r=0x7ff03cfffcf0) at ../ngx_lua-0.10.6/src/ngx_http_lua_util.c:569
  15. 14 ngx_http_lua_send_chain_link (r=0x7ff03cfffcf0, ctx=<optimized out>, in=0x0) at ../ngx_lua-0.10.6/src/ngx_http_lua_util.c:523
  16. 15 0x0000000000501f5f in ngx_http_lua_run_thread (L=L@entry=0x40030378, r=r@entry=0x7ff03cfffcf0, ctx=ctx@entry=0x7ff03d000dc8, nrets=nrets@entry=0) at ../ngx_lua-0.10.6/src/ngx_http_lua_util.c:1476
  17. 16 0x00000000005050f0 in ngx_http_lua_content_by_chunk (L=L@entry=0x40030378, r=r@entry=0x7ff03cfffcf0) at ../ngx_lua-0.10.6/src/ngx_http_lua_contentby.c:120
  18. 17 0x000000000050548d in ngx_http_lua_content_handler_file (r=0x7ff03cfffcf0) at ../ngx_lua-0.10.6/src/ngx_http_lua_contentby.c:284
  19. 18 0x0000000000504c21 in ngx_http_lua_content_handler (r=0x7ff03cfffcf0) at ../ngx_lua-0.10.6/src/ngx_http_lua_contentby.c:222
  20. 19 0x0000000000485e7f in ngx_http_core_content_phase (r=0x7ff03cfffcf0, ph=<optimized out>) at src/http/ngx_http_core_module.c:1379
  21. 20 0x0000000000480675 in ngx_http_core_run_phases (r=r@entry=0x7ff03cfffcf0) at src/http/ngx_http_core_module.c:856
  22. 21 0x000000000048075c in ngx_http_handler (r=r@entry=0x7ff03cfffcf0) at src/http/ngx_http_core_module.c:839
  23. 22 0x000000000048c249 in ngx_http_process_request (r=0x7ff03cfffcf0) at src/http/ngx_http_request.c:1916
  24. 23 0x000000000048cbf6 in ngx_http_process_request_line (rev=0x7ff03d069520) at src/http/ngx_http_request.c:1027
  25. 24 0x0000000000476029 in ngx_epoll_process_events (cycle=0x7ff03cffbce0, timer=<optimized out>, flags=<optimized out>) at src/event/modules/ngx_epoll_module.c:900
  26. 25 0x000000000046c947 in ngx_process_events_and_timers (cycle=cycle@entry=0x7ff03cffbce0) at src/event/ngx_event.c:242
  27. 26 0x0000000000473d35 in ngx_worker_process_cycle (cycle=cycle@entry=0x7ff03cffbce0, data=data@entry=0x0) at src/os/unix/ngx_process_cycle.c:753
  28. 27 0x0000000000472820 in ngx_spawn_process (cycle=cycle@entry=0x7ff03cffbce0, proc=0x473cf0 <ngx_worker_process_cycle>, data=0x0, name=0x6e22f5 "worker process", respawn=respawn@entry=0)
  29.    at src/os/unix/ngx_process.c:198
  30. 28 0x0000000000475216 in ngx_reap_children (cycle=0x7ff03cffbce0) at src/os/unix/ngx_process_cycle.c:621
  31. 29 ngx_master_process_cycle (cycle=cycle@entry=0x7ff03cffbce0) at src/os/unix/ngx_process_cycle.c:174
  32. 30 0x000000000044e139 in main (argc=<optimized out>, argv=<optimized out>) at src/core/nginx.c:367

断点

  1. 1       breakpoint     keep y   0x000000000048c420 in ngx_http_process_request_headers at src/http/ngx_http_request.c:1185
  2.         breakpoint already hit 13 times
  3. 2       breakpoint     keep y   0x0000000000504ba0 in ngx_http_lua_content_handler at ../ngx_lua-0.10.6/src/ngx_http_lua_contentby.c:154
  4.         breakpoint already hit 13 times
  5. 3       breakpoint     keep y   0x0000000000504c98 in ngx_http_lua_content_handler at ../ngx_lua-0.10.6/src/ngx_http_lua_contentby.c:189
  6. 4       breakpoint     keep y   0x0000000000504c19 in ngx_http_lua_content_handler at ../ngx_lua-0.10.6/src/ngx_http_lua_contentby.c:222
  7.         breakpoint already hit 8 times
  8. 5       breakpoint     keep y   0x0000000000505482 in ngx_http_lua_content_handler_file at ../ngx_lua-0.10.6/src/ngx_http_lua_contentby.c:284
  9.         breakpoint already hit 8 times
  10. 6       breakpoint     keep y   0x0000000000505024 in ngx_http_lua_content_by_chunk at ../ngx_lua-0.10.6/src/ngx_http_lua_contentby.c:54
  11.         breakpoint already hit 8 times
  12. 7       breakpoint     keep y   0x00000000005050e0 in ngx_http_lua_content_by_chunk at ../ngx_lua-0.10.6/src/ngx_http_lua_contentby.c:120
  13.         breakpoint already hit 8 times
  14. 8       breakpoint     keep y   0x00000000005014d3 in ngx_http_lua_run_thread at ../ngx_lua-0.10.6/src/ngx_http_lua_util.c:1005
  15.         breakpoint already hit 22 times
  16. 9       breakpoint     keep y   0x0000000000501f58 in ngx_http_lua_run_thread at ../ngx_lua-0.10.6/src/ngx_http_lua_util.c:1476
  17.         breakpoint already hit 8 times
  18. 10      breakpoint     keep y   0x00000000004ffc4a in ngx_http_lua_send_chain_link at ../ngx_lua-0.10.6/src/ngx_http_lua_util.c:484
  19. 11      breakpoint     keep y   0x00000000004ffb63 in ngx_http_lua_send_chain_link at ../ngx_lua-0.10.6/src/ngx_http_lua_util.c:557
  20.         breakpoint already hit 7 times
  21. 12      breakpoint     keep y   0x00000000004ffb86 in ngx_http_lua_send_chain_link at ../ngx_lua-0.10.6/src/ngx_http_lua_util.c:523
  22.         breakpoint already hit 6 times
  23. 13      breakpoint     keep y   0x00000000004ffc2b in ngx_http_lua_send_chain_link at ../ngx_lua-0.10.6/src/ngx_http_lua_util.c:569
  24.         breakpoint already hit 6 times
  25. 14      breakpoint     keep y   0x0000000000489e1b in ngx_http_send_special at src/http/ngx_http_request.c:3358
  26.         breakpoint already hit 6 times
  27. 15      breakpoint     keep y   0x0000000000485a2b in ngx_http_output_filter at src/http/ngx_http_core_module.c:1979
  28.         breakpoint already hit 30 times
  29. 16      breakpoint     keep y   0x0000000000506c70 in ngx_http_lua_capture_body_filter at ../ngx_lua-0.10.6/src/ngx_http_lua_capturefilter.c:133
  30.         breakpoint already hit 30 times
  31. 17      breakpoint     keep y   0x00000000004b2ce1 in ngx_http_charset_body_filter at src/http/modules/ngx_http_charset_filter_module.c:647
  32.         breakpoint already hit 30 times
  33. 18      breakpoint     keep y   0x00000000004afd89 in ngx_http_ssi_body_filter at src/http/modules/ngx_http_ssi_filter_module.c:411
  34.         breakpoint already hit 25 times
  35. 19      breakpoint     keep y   0x00000000004ad41d in ngx_http_postpone_filter at src/http/ngx_http_postpone_filter_module.c:82
  36.         breakpoint already hit 25 times
  37. 20      breakpoint     keep y   0x00000000004ac7d0 in ngx_http_gzip_body_filter at src/http/modules/ngx_http_gzip_filter_module.c:326
  38.         breakpoint already hit 25 times
  39. 21      breakpoint     keep y   0x00000000004a908f in ngx_http_chunked_body_filter at src/http/modules/ngx_http_chunked_filter_module.c:224
  40.         breakpoint already hit 25 times
  41. 22      breakpoint     keep y   0x00000000004a7cff in ngx_http_write_filter at src/http/ngx_http_write_filter_module.c:254
  42.         breakpoint already hit 5 times
  43. 23      breakpoint     keep y   0x00000000004769e1 in ngx_linux_sendfile_chain at src/os/unix/ngx_linux_sendfile_chain.c:92
  44.         breakpoint already hit 4 times
  45. 24      breakpoint     keep y   0x0000000000476957 in ngx_linux_sendfile_chain at src/os/unix/ngx_linux_sendfile_chain.c:49
  46.         breakpoint already hit 4 times
  47. 25      breakpoint     keep y   0x00000000004b4f8a in ngx_http_copy_filter at src/http/ngx_http_copy_filter_module.c:152
  48.         breakpoint already hit 5 times
  49. 26      breakpoint     keep y   0x000000000045338b in ngx_output_chain at src/core/ngx_output_chain.c:74
  50.         breakpoint already hit 5 times
  51. 27      breakpoint     keep y   0x0000000000476a71 in ngx_linux_sendfile_chain at src/os/unix/ngx_linux_sendfile_chain.c:215
  52.         breakpoint already hit 1 time
  53. 28      breakpoint     keep y   0x0000000000476a71 in ngx_linux_sendfile_chain at src/os/unix/ngx_linux_sendfile_chain.c:215
  54.         breakpoint already hit 1 time
  55. 29      breakpoint     keep y   0x0000000000476a71 in ngx_linux_sendfile_chain at src/os/unix/ngx_linux_sendfile_chain.c:21
  56. 30      breakpoint     keep y   0x0000000000476a71 in ngx_linux_sendfile_chain at src/os/unix/ngx_linux_sendfile_chain.c:215
  57.         breakpoint already hit 1 time
  58. 31      breakpoint     keep y   0x0000000000476a71 in ngx_linux_sendfile_chain at src/os/unix/ngx_linux_sendfile_chain.c:215
  59.         breakpoint already hit 1 time
  60. 32      breakpoint     keep y   0x0000000000476a71 in ngx_linux_sendfile_chain at src/os/unix/ngx_linux_sendfile_chain.c:215
  61.         breakpoint already hit 1 time
  62. 33      breakpoint     keep y   0x0000000000476a71 in ngx_linux_sendfile_chain at src/os/unix/ngx_linux_sendfile_chain.c:215
  63.         breakpoint already hit 1 time
  64. 34      breakpoint     keep y   0x0000000000476a71 in ngx_linux_sendfile_chain at src/os/unix/ngx_linux_sendfile_chain.c:215
  65.         breakpoint already hit 1 time
  66. 35      breakpoint     keep y   0x0000000000476a71 in ngx_linux_sendfile_chain at src/os/unix/ngx_linux_sendfile_chain.c:215
  67.         breakpoint already hit 1 time
  68. 36      breakpoint     keep y   0x0000000000476a71 in ngx_linux_sendfile_chain at src/os/unix/ngx_linux_sendfile_chain.c:215
  69.         breakpoint already hit 1 time

使用 GDB 调试 OpenResty

加断点

比如 ngx_http_lua_ffi_worker_pid

  1. local ffi = require 'ffi'
  2. local C = ffi.C
  5. ffi.cdef[[
  6. int ngx_http_lua_ffi_worker_pid(void);
  7. ]]
  9. local _M = { _VERSION = '0.10' }
  10. local mt = { __index = _M }
  12. function _M.new(self)
  13.     local _m_name = 'weibo.com'
  14.     return setmetatable({ m_name = _m_name }, mt)
  15. end
  17. function _M.getWorkerPid(self)
  18.     return C.ngx_http_lua_ffi_worker_pid()
  19. end
  21. return _M

运行调用这个包的代码，则可以在 GDB 中加断点 ngx_http_lua_ffi_worker_pid 即可直接追踪 OpenResty 的大致运行过程。通过 c 继续下一步

breakpoints

  1. ngx_http_process_request_headers
  2. ngx_http_lua_content_handler
  3. ngx_http_lua_content_handler_file
  4. ngx_http_lua_content_by_chunk
  5. ngx_http_lua_run_thread
  6. ngx_http_lua_send_chain_link
  7. ngx_http_send_special
  8. ngx_http_output_filter
  9. ngx_http_lua_capture_body_filter
  10. ngx_http_charset_body_filter
  11. ngx_http_ssi_body_filter
  12. ngx_http_postpone_filter
  13. ngx_http_gzip_body_filter
  14. ngx_http_chunked_body_filter
  15. ngx_http_write_filter
  16. ngx_linux_sendfile_chain
  17. ngx_http_copy_filter
  18. ngx_output_chain
  19. ngx_linux_sendfile_chain