system.stack_trace

Contains stack traces of all server threads. Allows developers to introspect the server state.

To analyze stack frames, use the addressToLine, addressToSymbol and demangle introspection functions.

Columns:

  • thread_id (UInt64) — Thread identifier.
  • query_id (String) — Query identifier that can be used to get details about a query that was running from the query_log system table.
  • trace (Array(UInt64)) — A stack trace which represents a list of physical addresses where the called methods are stored.

Example

Enabling introspection functions:

  1. SET allow_introspection_functions = 1;

Getting symbols from ClickHouse object files:

  1. WITH arrayMap(x -> demangle(addressToSymbol(x)), trace) AS all SELECT thread_id, query_id, arrayStringConcat(all, '\n') AS res FROM system.stack_trace LIMIT 1 \G
  1. Row 1:
  2. ──────
  3. thread_id: 686
  4. query_id:  1a11f70b-626d-47c1-b948-f9c7b206395d
  5. res:       sigqueue
  6. DB::StorageSystemStackTrace::fillData(std::__1::vector<COW<DB::IColumn>::mutable_ptr<DB::IColumn>, std::__1::allocator<COW<DB::IColumn>::mutable_ptr<DB::IColumn> > >&, DB::Context const&, DB::SelectQueryInfo const&) const
  7. DB::IStorageSystemOneBlock<DB::StorageSystemStackTrace>::read(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, DB::SelectQueryInfo const&, DB::Context const&, DB::QueryProcessingStage::Enum, unsigned long, unsigned int)
  8. DB::InterpreterSelectQuery::executeFetchColumns(DB::QueryProcessingStage::Enum, DB::QueryPipeline&, std::__1::shared_ptr<DB::PrewhereInfo> const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)
  9. DB::InterpreterSelectQuery::executeImpl(DB::QueryPipeline&, std::__1::shared_ptr<DB::IBlockInputStream> const&, std::__1::optional<DB::Pipe>)
  10. DB::InterpreterSelectQuery::execute()
  11. DB::InterpreterSelectWithUnionQuery::execute()
  12. DB::executeQueryImpl(char const*, char const*, DB::Context&, bool, DB::QueryProcessingStage::Enum, bool, DB::ReadBuffer*)
  13. DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::Context&, bool, DB::QueryProcessingStage::Enum, bool)
  14. DB::TCPHandler::runImpl()
  15. DB::TCPHandler::run()
  16. Poco::Net::TCPServerConnection::start()
  17. Poco::Net::TCPServerDispatcher::run()
  18. Poco::PooledThread::run()
  19. Poco::ThreadImpl::runnableEntry(void*)
  20. start_thread
  21. __clone

Getting filenames and line numbers in ClickHouse source code:

  1. WITH arrayMap(x -> addressToLine(x), trace) AS all, arrayFilter(x -> x LIKE '%/dbms/%', all) AS dbms SELECT thread_id, query_id, arrayStringConcat(notEmpty(dbms) ? dbms : all, '\n') AS res FROM system.stack_trace LIMIT 1 \G
  1. Row 1:
  2. ──────
  3. thread_id: 686
  4. query_id:  cad353e7-1c29-4b2e-949f-93e597ab7a54
  5. res:       /lib/x86_64-linux-gnu/libc-2.27.so
  6. /build/obj-x86_64-linux-gnu/../src/Storages/System/StorageSystemStackTrace.cpp:182
  7. /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/vector:656
  8. /build/obj-x86_64-linux-gnu/../src/Interpreters/InterpreterSelectQuery.cpp:1338
  9. /build/obj-x86_64-linux-gnu/../src/Interpreters/InterpreterSelectQuery.cpp:751
  10. /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/optional:224
  11. /build/obj-x86_64-linux-gnu/../src/Interpreters/InterpreterSelectWithUnionQuery.cpp:192
  12. /build/obj-x86_64-linux-gnu/../src/Interpreters/executeQuery.cpp:384
  13. /build/obj-x86_64-linux-gnu/../src/Interpreters/executeQuery.cpp:643
  14. /build/obj-x86_64-linux-gnu/../src/Server/TCPHandler.cpp:251
  15. /build/obj-x86_64-linux-gnu/../src/Server/TCPHandler.cpp:1197
  16. /build/obj-x86_64-linux-gnu/../contrib/poco/Net/src/TCPServerConnection.cpp:57
  17. /build/obj-x86_64-linux-gnu/../contrib/libcxx/include/atomic:856
  18. /build/obj-x86_64-linux-gnu/../contrib/poco/Foundation/include/Poco/Mutex_POSIX.h:59
  19. /build/obj-x86_64-linux-gnu/../contrib/poco/Foundation/include/Poco/AutoPtr.h:223
  20. /lib/x86_64-linux-gnu/libpthread-2.27.so
  21. /lib/x86_64-linux-gnu/libc-2.27.so

See Also

  • Introspection Functions — Which introspection functions are available and how to use them.
  • system.trace_log — Contains stack traces collected by the sampling query profiler.
  • arrayMap — Description and usage example of the arrayMap function.
  • arrayFilter — Description and usage example of the arrayFilter function.

Original article