Sampling Query Profiler

ClickHouse runs sampling profiler that allows analyzing query execution. Using profiler you can find source code routines that used the most frequently during query execution. You can trace CPU time and wall-clock time spent including idle time.

To use profiler:

  • Setup the trace_log section of the server configuration.

    This section configures the trace_log system table containing the results of the profiler functioning. It is configured by default. Remember that data in this table is valid only for a running server. After the server restart, ClickHouse doesn’t clean up the table and all the stored virtual memory address may become invalid.

  • Setup the query_profiler_cpu_time_period_ns or query_profiler_real_time_period_ns settings. Both settings can be used simultaneously.

    These settings allow you to configure profiler timers. As these are the session settings, you can get different sampling frequency for the whole server, individual users or user profiles, for your interactive session, and for each individual query.

The default sampling frequency is one sample per second and both CPU and real timers are enabled. This frequency allows collecting enough information about ClickHouse cluster. At the same time, working with this frequency, profiler doesn’t affect ClickHouse server’s performance. If you need to profile each individual query try to use higher sampling frequency.

To analyze the trace_log system table:

  • Install the clickhouse-common-static-dbg package. See Install from DEB Packages.

  • Allow introspection functions by the allow_introspection_functions setting.

    For security reasons, introspection functions are disabled by default.

  • Use the addressToLine, addressToSymbol and demangle introspection functions to get function names and their positions in ClickHouse code. To get a profile for some query, you need to aggregate data from the trace_log table. You can aggregate data by individual functions or by the whole stack traces.

If you need to visualize trace_log info, try flamegraph and speedscope.

Example

In this example we:

  • Filtering trace_log data by a query identifier and the current date.

  • Aggregating by stack trace.

  • Using introspection functions, we will get a report of:

    • Names of symbols and corresponding source code functions.
    • Source code locations of these functions.
  1. SELECT
  2.     count(),
  3.     arrayStringConcat(arrayMap(x -> concat(demangle(addressToSymbol(x)), '\n    ', addressToLine(x)), trace), '\n') AS sym
  4. FROM system.trace_log
  5. WHERE (query_id = 'ebca3574-ad0a-400a-9cbc-dca382f5998c') AND (event_date = today())
  6. GROUP BY trace
  7. ORDER BY count() DESC
  8. LIMIT 10
  1. Row 1:
  2. ──────
  3. count(): 6344
  4. sym:     StackTrace::StackTrace(ucontext_t const&)
  5.     /home/milovidov/ClickHouse/build_gcc9/../src/Common/StackTrace.cpp:208
  6. DB::(anonymous namespace)::writeTraceInfo(DB::TimerType, int, siginfo_t*, void*) [clone .isra.0]
  7.     /home/milovidov/ClickHouse/build_gcc9/../src/IO/BufferBase.h:99
  10. read
  12. DB::ReadBufferFromFileDescriptor::nextImpl()
  13.     /home/milovidov/ClickHouse/build_gcc9/../src/IO/ReadBufferFromFileDescriptor.cpp:56
  14. DB::CompressedReadBufferBase::readCompressedData(unsigned long&, unsigned long&)
  15.     /home/milovidov/ClickHouse/build_gcc9/../src/IO/ReadBuffer.h:54
  16. DB::CompressedReadBufferFromFile::nextImpl()
  17.     /home/milovidov/ClickHouse/build_gcc9/../src/Compression/CompressedReadBufferFromFile.cpp:22
  18. DB::CompressedReadBufferFromFile::seek(unsigned long, unsigned long)
  19.     /home/milovidov/ClickHouse/build_gcc9/../src/Compression/CompressedReadBufferFromFile.cpp:63
  20. DB::MergeTreeReaderStream::seekToMark(unsigned long)
  21.     /home/milovidov/ClickHouse/build_gcc9/../src/Storages/MergeTree/MergeTreeReaderStream.cpp:200
  22. std::_Function_handler<DB::ReadBuffer* (std::vector<DB::IDataType::Substream, std::allocator<DB::IDataType::Substream> > const&), DB::MergeTreeReader::readData(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, DB::IDataType const&, DB::IColumn&, unsigned long, bool, unsigned long, bool)::{lambda(bool)#1}::operator()(bool) const::{lambda(std::vector<DB::IDataType::Substream, std::allocator<DB::IDataType::Substream> > const&)#1}>::_M_invoke(std::_Any_data const&, std::vector<DB::IDataType::Substream, std::allocator<DB::IDataType::Substream> > const&)
  23.     /home/milovidov/ClickHouse/build_gcc9/../src/Storages/MergeTree/MergeTreeReader.cpp:212
  24. DB::IDataType::deserializeBinaryBulkWithMultipleStreams(DB::IColumn&, unsigned long, DB::IDataType::DeserializeBinaryBulkSettings&, std::shared_ptr<DB::IDataType::DeserializeBinaryBulkState>&) const
  25.     /usr/local/include/c++/9.1.0/bits/std_function.h:690
  26. DB::MergeTreeReader::readData(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, DB::IDataType const&, DB::IColumn&, unsigned long, bool, unsigned long, bool)
  27.     /home/milovidov/ClickHouse/build_gcc9/../src/Storages/MergeTree/MergeTreeReader.cpp:232
  28. DB::MergeTreeReader::readRows(unsigned long, bool, unsigned long, DB::Block&)
  29.     /home/milovidov/ClickHouse/build_gcc9/../src/Storages/MergeTree/MergeTreeReader.cpp:111
  30. DB::MergeTreeRangeReader::DelayedStream::finalize(DB::Block&)
  31.     /home/milovidov/ClickHouse/build_gcc9/../src/Storages/MergeTree/MergeTreeRangeReader.cpp:35
  32. DB::MergeTreeRangeReader::continueReadingChain(DB::MergeTreeRangeReader::ReadResult&)
  33.     /home/milovidov/ClickHouse/build_gcc9/../src/Storages/MergeTree/MergeTreeRangeReader.cpp:219
  34. DB::MergeTreeRangeReader::read(unsigned long, std::vector<DB::MarkRange, std::allocator<DB::MarkRange> >&)
  35.     /home/milovidov/ClickHouse/build_gcc9/../src/Storages/MergeTree/MergeTreeRangeReader.cpp:487
  36. DB::MergeTreeBaseSelectBlockInputStream::readFromPartImpl()
  37.     /home/milovidov/ClickHouse/build_gcc9/../src/Storages/MergeTree/MergeTreeBaseSelectBlockInputStream.cpp:158
  38. DB::MergeTreeBaseSelectBlockInputStream::readImpl()
  39.     /usr/local/include/c++/9.1.0/bits/stl_vector.h:108
  40. DB::IBlockInputStream::read()
  41.     /usr/local/include/c++/9.1.0/bits/stl_vector.h:108
  42. DB::ExpressionBlockInputStream::readImpl()
  43.     /home/milovidov/ClickHouse/build_gcc9/../src/DataStreams/ExpressionBlockInputStream.cpp:34
  44. DB::IBlockInputStream::read()
  45.     /usr/local/include/c++/9.1.0/bits/stl_vector.h:108
  46. DB::PartialSortingBlockInputStream::readImpl()
  47.     /home/milovidov/ClickHouse/build_gcc9/../src/DataStreams/PartialSortingBlockInputStream.cpp:13
  48. DB::IBlockInputStream::read()
  49.     /usr/local/include/c++/9.1.0/bits/stl_vector.h:108
  50. DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>::loop(unsigned long)
  51.     /usr/local/include/c++/9.1.0/bits/atomic_base.h:419
  52. DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>::thread(std::shared_ptr<DB::ThreadGroupStatus>, unsigned long)
  53.     /home/milovidov/ClickHouse/build_gcc9/../src/DataStreams/ParallelInputsProcessor.h:215
  54. ThreadFromGlobalPool::ThreadFromGlobalPool<void (DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>::*)(std::shared_ptr<DB::ThreadGroupStatus>, unsigned long), DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>*, std::shared_ptr<DB::ThreadGroupStatus>, unsigned long&>(void (DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>::*&&)(std::shared_ptr<DB::ThreadGroupStatus>, unsigned long), DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>*&&, std::shared_ptr<DB::ThreadGroupStatus>&&, unsigned long&)::{lambda()#1}::operator()() const
  55.     /usr/local/include/c++/9.1.0/bits/shared_ptr_base.h:729
  56. ThreadPoolImpl<std::thread>::worker(std::_List_iterator<std::thread>)
  57.     /usr/local/include/c++/9.1.0/bits/unique_lock.h:69
  58. execute_native_thread_routine
  59.     /home/milovidov/ClickHouse/ci/workspace/gcc/gcc-build/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/unique_ptr.h:81
  60. start_thread
  62. __clone
  65. Row 2:
  66. ──────
  67. count(): 3295
  68. sym:     StackTrace::StackTrace(ucontext_t const&)
  69.     /home/milovidov/ClickHouse/build_gcc9/../src/Common/StackTrace.cpp:208
  70. DB::(anonymous namespace)::writeTraceInfo(DB::TimerType, int, siginfo_t*, void*) [clone .isra.0]
  71.     /home/milovidov/ClickHouse/build_gcc9/../src/IO/BufferBase.h:99
  74. __pthread_cond_wait
  76. std::condition_variable::wait(std::unique_lock<std::mutex>&)
  77.     /home/milovidov/ClickHouse/ci/workspace/gcc/gcc-build/x86_64-pc-linux-gnu/libstdc++-v3/src/c++11/../../../../../gcc-9.1.0/libstdc++-v3/src/c++11/condition_variable.cc:55
  78. Poco::Semaphore::wait()
  79.     /home/milovidov/ClickHouse/build_gcc9/../contrib/poco/Foundation/src/Semaphore.cpp:61
  80. DB::UnionBlockInputStream::readImpl()
  81.     /usr/local/include/c++/9.1.0/x86_64-pc-linux-gnu/bits/gthr-default.h:748
  82. DB::IBlockInputStream::read()
  83.     /usr/local/include/c++/9.1.0/bits/stl_vector.h:108
  84. DB::MergeSortingBlockInputStream::readImpl()
  85.     /home/milovidov/ClickHouse/build_gcc9/../src/Core/Block.h:90
  86. DB::IBlockInputStream::read()
  87.     /usr/local/include/c++/9.1.0/bits/stl_vector.h:108
  88. DB::ExpressionBlockInputStream::readImpl()
  89.     /home/milovidov/ClickHouse/build_gcc9/../src/DataStreams/ExpressionBlockInputStream.cpp:34
  90. DB::IBlockInputStream::read()
  91.     /usr/local/include/c++/9.1.0/bits/stl_vector.h:108
  92. DB::LimitBlockInputStream::readImpl()
  93.     /usr/local/include/c++/9.1.0/bits/stl_vector.h:108
  94. DB::IBlockInputStream::read()
  95.     /usr/local/include/c++/9.1.0/bits/stl_vector.h:108
  96. DB::AsynchronousBlockInputStream::calculate()
  97.     /usr/local/include/c++/9.1.0/bits/stl_vector.h:108
  98. std::_Function_handler<void (), DB::AsynchronousBlockInputStream::next()::{lambda()#1}>::_M_invoke(std::_Any_data const&)
  99.     /usr/local/include/c++/9.1.0/bits/atomic_base.h:551
  100. ThreadPoolImpl<ThreadFromGlobalPool>::worker(std::_List_iterator<ThreadFromGlobalPool>)
  101.     /usr/local/include/c++/9.1.0/x86_64-pc-linux-gnu/bits/gthr-default.h:748
  102. ThreadFromGlobalPool::ThreadFromGlobalPool<ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::function<void ()>, int, std::optional<unsigned long>)::{lambda()#3}>(ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::function<void ()>, int, std::optional<unsigned long>)::{lambda()#3}&&)::{lambda()#1}::operator()() const
  103.     /home/milovidov/ClickHouse/build_gcc9/../src/Common/ThreadPool.h:146
  104. ThreadPoolImpl<std::thread>::worker(std::_List_iterator<std::thread>)
  105.     /usr/local/include/c++/9.1.0/bits/unique_lock.h:69
  106. execute_native_thread_routine
  107.     /home/milovidov/ClickHouse/ci/workspace/gcc/gcc-build/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/unique_ptr.h:81
  108. start_thread
  110. __clone
  113. Row 3:
  114. ──────
  115. count(): 1978
  116. sym:     StackTrace::StackTrace(ucontext_t const&)
  117.     /home/milovidov/ClickHouse/build_gcc9/../src/Common/StackTrace.cpp:208
  118. DB::(anonymous namespace)::writeTraceInfo(DB::TimerType, int, siginfo_t*, void*) [clone .isra.0]
  119.     /home/milovidov/ClickHouse/build_gcc9/../src/IO/BufferBase.h:99
  122. DB::VolnitskyBase<true, true, DB::StringSearcher<true, true> >::search(unsigned char const*, unsigned long) const
  123.     /opt/milovidov/ClickHouse/build_gcc9/programs/clickhouse
  124. DB::MatchImpl<true, false>::vector_constant(DB::PODArray<unsigned char, 4096ul, AllocatorWithHint<false, AllocatorHints::DefaultHint, 67108864ul>, 15ul, 16ul> const&, DB::PODArray<unsigned long, 4096ul, AllocatorWithHint<false, AllocatorHints::DefaultHint, 67108864ul>, 15ul, 16ul> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, DB::PODArray<unsigned char, 4096ul, AllocatorWithHint<false, AllocatorHints::DefaultHint, 67108864ul>, 15ul, 16ul>&)
  125.     /opt/milovidov/ClickHouse/build_gcc9/programs/clickhouse
  126. DB::FunctionsStringSearch<DB::MatchImpl<true, false>, DB::NameLike>::executeImpl(DB::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long)
  127.     /opt/milovidov/ClickHouse/build_gcc9/programs/clickhouse
  128. DB::PreparedFunctionImpl::execute(DB::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long, bool)
  129.     /home/milovidov/ClickHouse/build_gcc9/../src/Functions/IFunction.cpp:464
  130. DB::ExpressionAction::execute(DB::Block&, bool) const
  131.     /usr/local/include/c++/9.1.0/bits/stl_vector.h:677
  132. DB::ExpressionActions::execute(DB::Block&, bool) const
  133.     /home/milovidov/ClickHouse/build_gcc9/../src/Interpreters/ExpressionActions.cpp:739
  134. DB::MergeTreeRangeReader::executePrewhereActionsAndFilterColumns(DB::MergeTreeRangeReader::ReadResult&)
  135.     /home/milovidov/ClickHouse/build_gcc9/../src/Storages/MergeTree/MergeTreeRangeReader.cpp:660
  136. DB::MergeTreeRangeReader::read(unsigned long, std::vector<DB::MarkRange, std::allocator<DB::MarkRange> >&)
  137.     /home/milovidov/ClickHouse/build_gcc9/../src/Storages/MergeTree/MergeTreeRangeReader.cpp:546
  138. DB::MergeTreeRangeReader::read(unsigned long, std::vector<DB::MarkRange, std::allocator<DB::MarkRange> >&)
  139.     /usr/local/include/c++/9.1.0/bits/stl_vector.h:108
  140. DB::MergeTreeBaseSelectBlockInputStream::readFromPartImpl()
  141.     /home/milovidov/ClickHouse/build_gcc9/../src/Storages/MergeTree/MergeTreeBaseSelectBlockInputStream.cpp:158
  142. DB::MergeTreeBaseSelectBlockInputStream::readImpl()
  143.     /usr/local/include/c++/9.1.0/bits/stl_vector.h:108
  144. DB::IBlockInputStream::read()
  145.     /usr/local/include/c++/9.1.0/bits/stl_vector.h:108
  146. DB::ExpressionBlockInputStream::readImpl()
  147.     /home/milovidov/ClickHouse/build_gcc9/../src/DataStreams/ExpressionBlockInputStream.cpp:34
  148. DB::IBlockInputStream::read()
  149.     /usr/local/include/c++/9.1.0/bits/stl_vector.h:108
  150. DB::PartialSortingBlockInputStream::readImpl()
  151.     /home/milovidov/ClickHouse/build_gcc9/../src/DataStreams/PartialSortingBlockInputStream.cpp:13
  152. DB::IBlockInputStream::read()
  153.     /usr/local/include/c++/9.1.0/bits/stl_vector.h:108
  154. DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>::loop(unsigned long)
  155.     /usr/local/include/c++/9.1.0/bits/atomic_base.h:419
  156. DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>::thread(std::shared_ptr<DB::ThreadGroupStatus>, unsigned long)
  157.     /home/milovidov/ClickHouse/build_gcc9/../src/DataStreams/ParallelInputsProcessor.h:215
  158. ThreadFromGlobalPool::ThreadFromGlobalPool<void (DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>::*)(std::shared_ptr<DB::ThreadGroupStatus>, unsigned long), DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>*, std::shared_ptr<DB::ThreadGroupStatus>, unsigned long&>(void (DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>::*&&)(std::shared_ptr<DB::ThreadGroupStatus>, unsigned long), DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>*&&, std::shared_ptr<DB::ThreadGroupStatus>&&, unsigned long&)::{lambda()#1}::operator()() const
  159.     /usr/local/include/c++/9.1.0/bits/shared_ptr_base.h:729
  160. ThreadPoolImpl<std::thread>::worker(std::_List_iterator<std::thread>)
  161.     /usr/local/include/c++/9.1.0/bits/unique_lock.h:69
  162. execute_native_thread_routine
  163.     /home/milovidov/ClickHouse/ci/workspace/gcc/gcc-build/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/unique_ptr.h:81
  164. start_thread
  166. __clone
  169. Row 4:
  170. ──────
  171. count(): 1913
  172. sym:     StackTrace::StackTrace(ucontext_t const&)
  173.     /home/milovidov/ClickHouse/build_gcc9/../src/Common/StackTrace.cpp:208
  174. DB::(anonymous namespace)::writeTraceInfo(DB::TimerType, int, siginfo_t*, void*) [clone .isra.0]
  175.     /home/milovidov/ClickHouse/build_gcc9/../src/IO/BufferBase.h:99
  178. DB::VolnitskyBase<true, true, DB::StringSearcher<true, true> >::search(unsigned char const*, unsigned long) const
  179.     /opt/milovidov/ClickHouse/build_gcc9/programs/clickhouse
  180. DB::MatchImpl<true, false>::vector_constant(DB::PODArray<unsigned char, 4096ul, AllocatorWithHint<false, AllocatorHints::DefaultHint, 67108864ul>, 15ul, 16ul> const&, DB::PODArray<unsigned long, 4096ul, AllocatorWithHint<false, AllocatorHints::DefaultHint, 67108864ul>, 15ul, 16ul> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, DB::PODArray<unsigned char, 4096ul, AllocatorWithHint<false, AllocatorHints::DefaultHint, 67108864ul>, 15ul, 16ul>&)
  181.     /opt/milovidov/ClickHouse/build_gcc9/programs/clickhouse
  182. DB::FunctionsStringSearch<DB::MatchImpl<true, false>, DB::NameLike>::executeImpl(DB::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long)
  183.     /opt/milovidov/ClickHouse/build_gcc9/programs/clickhouse
  184. DB::PreparedFunctionImpl::execute(DB::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long, bool)
  185.     /home/milovidov/ClickHouse/build_gcc9/../src/Functions/IFunction.cpp:464
  186. DB::ExpressionAction::execute(DB::Block&, bool) const
  187.     /usr/local/include/c++/9.1.0/bits/stl_vector.h:677
  188. DB::ExpressionActions::execute(DB::Block&, bool) const
  189.     /home/milovidov/ClickHouse/build_gcc9/../src/Interpreters/ExpressionActions.cpp:739
  190. DB::MergeTreeRangeReader::executePrewhereActionsAndFilterColumns(DB::MergeTreeRangeReader::ReadResult&)
  191.     /home/milovidov/ClickHouse/build_gcc9/../src/Storages/MergeTree/MergeTreeRangeReader.cpp:660
  192. DB::MergeTreeRangeReader::read(unsigned long, std::vector<DB::MarkRange, std::allocator<DB::MarkRange> >&)
  193.     /home/milovidov/ClickHouse/build_gcc9/../src/Storages/MergeTree/MergeTreeRangeReader.cpp:546
  194. DB::MergeTreeRangeReader::read(unsigned long, std::vector<DB::MarkRange, std::allocator<DB::MarkRange> >&)
  195.     /usr/local/include/c++/9.1.0/bits/stl_vector.h:108
  196. DB::MergeTreeBaseSelectBlockInputStream::readFromPartImpl()
  197.     /home/milovidov/ClickHouse/build_gcc9/../src/Storages/MergeTree/MergeTreeBaseSelectBlockInputStream.cpp:158
  198. DB::MergeTreeBaseSelectBlockInputStream::readImpl()
  199.     /usr/local/include/c++/9.1.0/bits/stl_vector.h:108
  200. DB::IBlockInputStream::read()
  201.     /usr/local/include/c++/9.1.0/bits/stl_vector.h:108
  202. DB::ExpressionBlockInputStream::readImpl()
  203.     /home/milovidov/ClickHouse/build_gcc9/../src/DataStreams/ExpressionBlockInputStream.cpp:34
  204. DB::IBlockInputStream::read()
  205.     /usr/local/include/c++/9.1.0/bits/stl_vector.h:108
  206. DB::PartialSortingBlockInputStream::readImpl()
  207.     /home/milovidov/ClickHouse/build_gcc9/../src/DataStreams/PartialSortingBlockInputStream.cpp:13
  208. DB::IBlockInputStream::read()
  209.     /usr/local/include/c++/9.1.0/bits/stl_vector.h:108
  210. DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>::loop(unsigned long)
  211.     /usr/local/include/c++/9.1.0/bits/atomic_base.h:419
  212. DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>::thread(std::shared_ptr<DB::ThreadGroupStatus>, unsigned long)
  213.     /home/milovidov/ClickHouse/build_gcc9/../src/DataStreams/ParallelInputsProcessor.h:215
  214. ThreadFromGlobalPool::ThreadFromGlobalPool<void (DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>::*)(std::shared_ptr<DB::ThreadGroupStatus>, unsigned long), DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>*, std::shared_ptr<DB::ThreadGroupStatus>, unsigned long&>(void (DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>::*&&)(std::shared_ptr<DB::ThreadGroupStatus>, unsigned long), DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>*&&, std::shared_ptr<DB::ThreadGroupStatus>&&, unsigned long&)::{lambda()#1}::operator()() const
  215.     /usr/local/include/c++/9.1.0/bits/shared_ptr_base.h:729
  216. ThreadPoolImpl<std::thread>::worker(std::_List_iterator<std::thread>)
  217.     /usr/local/include/c++/9.1.0/bits/unique_lock.h:69
  218. execute_native_thread_routine
  219.     /home/milovidov/ClickHouse/ci/workspace/gcc/gcc-build/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/unique_ptr.h:81
  220. start_thread
  222. __clone
  225. Row 5:
  226. ──────
  227. count(): 1672
  228. sym:     StackTrace::StackTrace(ucontext_t const&)
  229.     /home/milovidov/ClickHouse/build_gcc9/../src/Common/StackTrace.cpp:208
  230. DB::(anonymous namespace)::writeTraceInfo(DB::TimerType, int, siginfo_t*, void*) [clone .isra.0]
  231.     /home/milovidov/ClickHouse/build_gcc9/../src/IO/BufferBase.h:99
  234. DB::VolnitskyBase<true, true, DB::StringSearcher<true, true> >::search(unsigned char const*, unsigned long) const
  235.     /opt/milovidov/ClickHouse/build_gcc9/programs/clickhouse
  236. DB::MatchImpl<true, false>::vector_constant(DB::PODArray<unsigned char, 4096ul, AllocatorWithHint<false, AllocatorHints::DefaultHint, 67108864ul>, 15ul, 16ul> const&, DB::PODArray<unsigned long, 4096ul, AllocatorWithHint<false, AllocatorHints::DefaultHint, 67108864ul>, 15ul, 16ul> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, DB::PODArray<unsigned char, 4096ul, AllocatorWithHint<false, AllocatorHints::DefaultHint, 67108864ul>, 15ul, 16ul>&)
  237.     /opt/milovidov/ClickHouse/build_gcc9/programs/clickhouse
  238. DB::FunctionsStringSearch<DB::MatchImpl<true, false>, DB::NameLike>::executeImpl(DB::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long)
  239.     /opt/milovidov/ClickHouse/build_gcc9/programs/clickhouse
  240. DB::PreparedFunctionImpl::execute(DB::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long, bool)
  241.     /home/milovidov/ClickHouse/build_gcc9/../src/Functions/IFunction.cpp:464
  242. DB::ExpressionAction::execute(DB::Block&, bool) const
  243.     /usr/local/include/c++/9.1.0/bits/stl_vector.h:677
  244. DB::ExpressionActions::execute(DB::Block&, bool) const
  245.     /home/milovidov/ClickHouse/build_gcc9/../src/Interpreters/ExpressionActions.cpp:739
  246. DB::MergeTreeRangeReader::executePrewhereActionsAndFilterColumns(DB::MergeTreeRangeReader::ReadResult&)
  247.     /home/milovidov/ClickHouse/build_gcc9/../src/Storages/MergeTree/MergeTreeRangeReader.cpp:660
  248. DB::MergeTreeRangeReader::read(unsigned long, std::vector<DB::MarkRange, std::allocator<DB::MarkRange> >&)
  249.     /home/milovidov/ClickHouse/build_gcc9/../src/Storages/MergeTree/MergeTreeRangeReader.cpp:546
  250. DB::MergeTreeRangeReader::read(unsigned long, std::vector<DB::MarkRange, std::allocator<DB::MarkRange> >&)
  251.     /usr/local/include/c++/9.1.0/bits/stl_vector.h:108
  252. DB::MergeTreeBaseSelectBlockInputStream::readFromPartImpl()
  253.     /home/milovidov/ClickHouse/build_gcc9/../src/Storages/MergeTree/MergeTreeBaseSelectBlockInputStream.cpp:158
  254. DB::MergeTreeBaseSelectBlockInputStream::readImpl()
  255.     /usr/local/include/c++/9.1.0/bits/stl_vector.h:108
  256. DB::IBlockInputStream::read()
  257.     /usr/local/include/c++/9.1.0/bits/stl_vector.h:108
  258. DB::ExpressionBlockInputStream::readImpl()
  259.     /home/milovidov/ClickHouse/build_gcc9/../src/DataStreams/ExpressionBlockInputStream.cpp:34
  260. DB::IBlockInputStream::read()
  261.     /usr/local/include/c++/9.1.0/bits/stl_vector.h:108
  262. DB::PartialSortingBlockInputStream::readImpl()
  263.     /home/milovidov/ClickHouse/build_gcc9/../src/DataStreams/PartialSortingBlockInputStream.cpp:13
  264. DB::IBlockInputStream::read()
  265.     /usr/local/include/c++/9.1.0/bits/stl_vector.h:108
  266. DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>::loop(unsigned long)
  267.     /usr/local/include/c++/9.1.0/bits/atomic_base.h:419
  268. DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>::thread(std::shared_ptr<DB::ThreadGroupStatus>, unsigned long)
  269.     /home/milovidov/ClickHouse/build_gcc9/../src/DataStreams/ParallelInputsProcessor.h:215
  270. ThreadFromGlobalPool::ThreadFromGlobalPool<void (DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>::*)(std::shared_ptr<DB::ThreadGroupStatus>, unsigned long), DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>*, std::shared_ptr<DB::ThreadGroupStatus>, unsigned long&>(void (DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>::*&&)(std::shared_ptr<DB::ThreadGroupStatus>, unsigned long), DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>*&&, std::shared_ptr<DB::ThreadGroupStatus>&&, unsigned long&)::{lambda()#1}::operator()() const
  271.     /usr/local/include/c++/9.1.0/bits/shared_ptr_base.h:729
  272. ThreadPoolImpl<std::thread>::worker(std::_List_iterator<std::thread>)
  273.     /usr/local/include/c++/9.1.0/bits/unique_lock.h:69
  274. execute_native_thread_routine
  275.     /home/milovidov/ClickHouse/ci/workspace/gcc/gcc-build/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/unique_ptr.h:81
  276. start_thread
  278. __clone
  281. Row 6:
  282. ──────
  283. count(): 1531
  284. sym:     StackTrace::StackTrace(ucontext_t const&)
  285.     /home/milovidov/ClickHouse/build_gcc9/../src/Common/StackTrace.cpp:208
  286. DB::(anonymous namespace)::writeTraceInfo(DB::TimerType, int, siginfo_t*, void*) [clone .isra.0]
  287.     /home/milovidov/ClickHouse/build_gcc9/../src/IO/BufferBase.h:99
  290. read
  292. DB::ReadBufferFromFileDescriptor::nextImpl()
  293.     /home/milovidov/ClickHouse/build_gcc9/../src/IO/ReadBufferFromFileDescriptor.cpp:56
  294. DB::CompressedReadBufferBase::readCompressedData(unsigned long&, unsigned long&)
  295.     /home/milovidov/ClickHouse/build_gcc9/../src/IO/ReadBuffer.h:54
  296. DB::CompressedReadBufferFromFile::nextImpl()
  297.     /home/milovidov/ClickHouse/build_gcc9/../src/Compression/CompressedReadBufferFromFile.cpp:22
  298. void DB::deserializeBinarySSE2<4>(DB::PODArray<unsigned char, 4096ul, AllocatorWithHint<false, AllocatorHints::DefaultHint, 67108864ul>, 15ul, 16ul>&, DB::PODArray<unsigned long, 4096ul, AllocatorWithHint<false, AllocatorHints::DefaultHint, 67108864ul>, 15ul, 16ul>&, DB::ReadBuffer&, unsigned long)
  299.     /home/milovidov/ClickHouse/build_gcc9/../src/IO/ReadBuffer.h:53
  300. DB::DataTypeString::deserializeBinaryBulk(DB::IColumn&, DB::ReadBuffer&, unsigned long, double) const
  301.     /home/milovidov/ClickHouse/build_gcc9/../src/DataTypes/DataTypeString.cpp:202
  302. DB::MergeTreeReader::readData(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, DB::IDataType const&, DB::IColumn&, unsigned long, bool, unsigned long, bool)
  303.     /home/milovidov/ClickHouse/build_gcc9/../src/Storages/MergeTree/MergeTreeReader.cpp:232
  304. DB::MergeTreeReader::readRows(unsigned long, bool, unsigned long, DB::Block&)
  305.     /home/milovidov/ClickHouse/build_gcc9/../src/Storages/MergeTree/MergeTreeReader.cpp:111
  306. DB::MergeTreeRangeReader::DelayedStream::finalize(DB::Block&)
  307.     /home/milovidov/ClickHouse/build_gcc9/../src/Storages/MergeTree/MergeTreeRangeReader.cpp:35
  308. DB::MergeTreeRangeReader::startReadingChain(unsigned long, std::vector<DB::MarkRange, std::allocator<DB::MarkRange> >&)
  309.     /home/milovidov/ClickHouse/build_gcc9/../src/Storages/MergeTree/MergeTreeRangeReader.cpp:219
  310. DB::MergeTreeRangeReader::read(unsigned long, std::vector<DB::MarkRange, std::allocator<DB::MarkRange> >&)
  311.     /usr/local/include/c++/9.1.0/bits/stl_vector.h:108
  312. DB::MergeTreeRangeReader::read(unsigned long, std::vector<DB::MarkRange, std::allocator<DB::MarkRange> >&)
  313.     /usr/local/include/c++/9.1.0/bits/stl_vector.h:108
  314. DB::MergeTreeBaseSelectBlockInputStream::readFromPartImpl()
  315.     /home/milovidov/ClickHouse/build_gcc9/../src/Storages/MergeTree/MergeTreeBaseSelectBlockInputStream.cpp:158
  316. DB::MergeTreeBaseSelectBlockInputStream::readImpl()
  317.     /usr/local/include/c++/9.1.0/bits/stl_vector.h:108
  318. DB::IBlockInputStream::read()
  319.     /usr/local/include/c++/9.1.0/bits/stl_vector.h:108
  320. DB::ExpressionBlockInputStream::readImpl()
  321.     /home/milovidov/ClickHouse/build_gcc9/../src/DataStreams/ExpressionBlockInputStream.cpp:34
  322. DB::IBlockInputStream::read()
  323.     /usr/local/include/c++/9.1.0/bits/stl_vector.h:108
  324. DB::PartialSortingBlockInputStream::readImpl()
  325.     /home/milovidov/ClickHouse/build_gcc9/../src/DataStreams/PartialSortingBlockInputStream.cpp:13
  326. DB::IBlockInputStream::read()
  327.     /usr/local/include/c++/9.1.0/bits/stl_vector.h:108
  328. DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>::loop(unsigned long)
  329.     /usr/local/include/c++/9.1.0/bits/atomic_base.h:419
  330. DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>::thread(std::shared_ptr<DB::ThreadGroupStatus>, unsigned long)
  331.     /home/milovidov/ClickHouse/build_gcc9/../src/DataStreams/ParallelInputsProcessor.h:215
  332. ThreadFromGlobalPool::ThreadFromGlobalPool<void (DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>::*)(std::shared_ptr<DB::ThreadGroupStatus>, unsigned long), DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>*, std::shared_ptr<DB::ThreadGroupStatus>, unsigned long&>(void (DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>::*&&)(std::shared_ptr<DB::ThreadGroupStatus>, unsigned long), DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>*&&, std::shared_ptr<DB::ThreadGroupStatus>&&, unsigned long&)::{lambda()#1}::operator()() const
  333.     /usr/local/include/c++/9.1.0/bits/shared_ptr_base.h:729
  334. ThreadPoolImpl<std::thread>::worker(std::_List_iterator<std::thread>)
  335.     /usr/local/include/c++/9.1.0/bits/unique_lock.h:69
  336. execute_native_thread_routine
  337.     /home/milovidov/ClickHouse/ci/workspace/gcc/gcc-build/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/unique_ptr.h:81
  338. start_thread
  340. __clone
  343. Row 7:
  344. ──────
  345. count(): 1034
  346. sym:     StackTrace::StackTrace(ucontext_t const&)
  347.     /home/milovidov/ClickHouse/build_gcc9/../src/Common/StackTrace.cpp:208
  348. DB::(anonymous namespace)::writeTraceInfo(DB::TimerType, int, siginfo_t*, void*) [clone .isra.0]
  349.     /home/milovidov/ClickHouse/build_gcc9/../src/IO/BufferBase.h:99
  352. DB::VolnitskyBase<true, true, DB::StringSearcher<true, true> >::search(unsigned char const*, unsigned long) const
  353.     /opt/milovidov/ClickHouse/build_gcc9/programs/clickhouse
  354. DB::MatchImpl<true, false>::vector_constant(DB::PODArray<unsigned char, 4096ul, AllocatorWithHint<false, AllocatorHints::DefaultHint, 67108864ul>, 15ul, 16ul> const&, DB::PODArray<unsigned long, 4096ul, AllocatorWithHint<false, AllocatorHints::DefaultHint, 67108864ul>, 15ul, 16ul> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, DB::PODArray<unsigned char, 4096ul, AllocatorWithHint<false, AllocatorHints::DefaultHint, 67108864ul>, 15ul, 16ul>&)
  355.     /opt/milovidov/ClickHouse/build_gcc9/programs/clickhouse
  356. DB::FunctionsStringSearch<DB::MatchImpl<true, false>, DB::NameLike>::executeImpl(DB::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long)
  357.     /opt/milovidov/ClickHouse/build_gcc9/programs/clickhouse
  358. DB::PreparedFunctionImpl::execute(DB::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long, bool)
  359.     /home/milovidov/ClickHouse/build_gcc9/../src/Functions/IFunction.cpp:464
  360. DB::ExpressionAction::execute(DB::Block&, bool) const
  361.     /usr/local/include/c++/9.1.0/bits/stl_vector.h:677
  362. DB::ExpressionActions::execute(DB::Block&, bool) const
  363.     /home/milovidov/ClickHouse/build_gcc9/../src/Interpreters/ExpressionActions.cpp:739
  364. DB::MergeTreeRangeReader::executePrewhereActionsAndFilterColumns(DB::MergeTreeRangeReader::ReadResult&)
  365.     /home/milovidov/ClickHouse/build_gcc9/../src/Storages/MergeTree/MergeTreeRangeReader.cpp:660
  366. DB::MergeTreeRangeReader::read(unsigned long, std::vector<DB::MarkRange, std::allocator<DB::MarkRange> >&)
  367.     /home/milovidov/ClickHouse/build_gcc9/../src/Storages/MergeTree/MergeTreeRangeReader.cpp:546
  368. DB::MergeTreeRangeReader::read(unsigned long, std::vector<DB::MarkRange, std::allocator<DB::MarkRange> >&)
  369.     /usr/local/include/c++/9.1.0/bits/stl_vector.h:108
  370. DB::MergeTreeBaseSelectBlockInputStream::readFromPartImpl()
  371.     /home/milovidov/ClickHouse/build_gcc9/../src/Storages/MergeTree/MergeTreeBaseSelectBlockInputStream.cpp:158
  372. DB::MergeTreeBaseSelectBlockInputStream::readImpl()
  373.     /usr/local/include/c++/9.1.0/bits/stl_vector.h:108
  374. DB::IBlockInputStream::read()
  375.     /usr/local/include/c++/9.1.0/bits/stl_vector.h:108
  376. DB::ExpressionBlockInputStream::readImpl()
  377.     /home/milovidov/ClickHouse/build_gcc9/../src/DataStreams/ExpressionBlockInputStream.cpp:34
  378. DB::IBlockInputStream::read()
  379.     /usr/local/include/c++/9.1.0/bits/stl_vector.h:108
  380. DB::PartialSortingBlockInputStream::readImpl()
  381.     /home/milovidov/ClickHouse/build_gcc9/../src/DataStreams/PartialSortingBlockInputStream.cpp:13
  382. DB::IBlockInputStream::read()
  383.     /usr/local/include/c++/9.1.0/bits/stl_vector.h:108
  384. DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>::loop(unsigned long)
  385.     /usr/local/include/c++/9.1.0/bits/atomic_base.h:419
  386. DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>::thread(std::shared_ptr<DB::ThreadGroupStatus>, unsigned long)
  387.     /home/milovidov/ClickHouse/build_gcc9/../src/DataStreams/ParallelInputsProcessor.h:215
  388. ThreadFromGlobalPool::ThreadFromGlobalPool<void (DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>::*)(std::shared_ptr<DB::ThreadGroupStatus>, unsigned long), DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>*, std::shared_ptr<DB::ThreadGroupStatus>, unsigned long&>(void (DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>::*&&)(std::shared_ptr<DB::ThreadGroupStatus>, unsigned long), DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>*&&, std::shared_ptr<DB::ThreadGroupStatus>&&, unsigned long&)::{lambda()#1}::operator()() const
  389.     /usr/local/include/c++/9.1.0/bits/shared_ptr_base.h:729
  390. ThreadPoolImpl<std::thread>::worker(std::_List_iterator<std::thread>)
  391.     /usr/local/include/c++/9.1.0/bits/unique_lock.h:69
  392. execute_native_thread_routine
  393.     /home/milovidov/ClickHouse/ci/workspace/gcc/gcc-build/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/unique_ptr.h:81
  394. start_thread
  396. __clone
  399. Row 8:
  400. ──────
  401. count(): 989
  402. sym:     StackTrace::StackTrace(ucontext_t const&)
  403.     /home/milovidov/ClickHouse/build_gcc9/../src/Common/StackTrace.cpp:208
  404. DB::(anonymous namespace)::writeTraceInfo(DB::TimerType, int, siginfo_t*, void*) [clone .isra.0]
  405.     /home/milovidov/ClickHouse/build_gcc9/../src/IO/BufferBase.h:99
  408. __lll_lock_wait
  410. pthread_mutex_lock
  412. DB::MergeTreeReaderStream::loadMarks()
  413.     /usr/local/include/c++/9.1.0/bits/std_mutex.h:103
  414. DB::MergeTreeReaderStream::MergeTreeReaderStream(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, std::vector<DB::MarkRange, std::allocator<DB::MarkRange> > const&, DB::MarkCache*, bool, DB::UncompressedCache*, unsigned long, unsigned long, unsigned long, DB::MergeTreeIndexGranularityInfo const*, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)
  415.     /home/milovidov/ClickHouse/build_gcc9/../src/Storages/MergeTree/MergeTreeReaderStream.cpp:107
  416. std::_Function_handler<void (std::vector<DB::IDataType::Substream, std::allocator<DB::IDataType::Substream> > const&), DB::MergeTreeReader::addStreams(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, DB::IDataType const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)::{lambda(std::vector<DB::IDataType::Substream, std::allocator<DB::IDataType::Substream> > const&)#1}>::_M_invoke(std::_Any_data const&, std::vector<DB::IDataType::Substream, std::allocator<DB::IDataType::Substream> > const&)
  417.     /usr/local/include/c++/9.1.0/bits/unique_ptr.h:147
  418. DB::MergeTreeReader::addStreams(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, DB::IDataType const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)
  419.     /usr/local/include/c++/9.1.0/bits/stl_vector.h:677
  420. DB::MergeTreeReader::MergeTreeReader(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<DB::MergeTreeDataPart const> const&, DB::NamesAndTypesList const&, DB::UncompressedCache*, DB::MarkCache*, bool, DB::MergeTreeData const&, std::vector<DB::MarkRange, std::allocator<DB::MarkRange> > const&, unsigned long, unsigned long, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, double, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, double> > > const&, std::function<void (DB::ReadBufferFromFileBase::ProfileInfo)> const&, int)
  421.     /usr/local/include/c++/9.1.0/bits/stl_list.h:303
  422. DB::MergeTreeThreadSelectBlockInputStream::getNewTask()
  423.     /usr/local/include/c++/9.1.0/bits/std_function.h:259
  424. DB::MergeTreeBaseSelectBlockInputStream::readImpl()
  425.     /home/milovidov/ClickHouse/build_gcc9/../src/Storages/MergeTree/MergeTreeBaseSelectBlockInputStream.cpp:54
  426. DB::IBlockInputStream::read()
  427.     /usr/local/include/c++/9.1.0/bits/stl_vector.h:108
  428. DB::ExpressionBlockInputStream::readImpl()
  429.     /home/milovidov/ClickHouse/build_gcc9/../src/DataStreams/ExpressionBlockInputStream.cpp:34
  430. DB::IBlockInputStream::read()
  431.     /usr/local/include/c++/9.1.0/bits/stl_vector.h:108
  432. DB::PartialSortingBlockInputStream::readImpl()
  433.     /home/milovidov/ClickHouse/build_gcc9/../src/DataStreams/PartialSortingBlockInputStream.cpp:13
  434. DB::IBlockInputStream::read()
  435.     /usr/local/include/c++/9.1.0/bits/stl_vector.h:108
  436. DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>::loop(unsigned long)
  437.     /usr/local/include/c++/9.1.0/bits/atomic_base.h:419
  438. DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>::thread(std::shared_ptr<DB::ThreadGroupStatus>, unsigned long)
  439.     /home/milovidov/ClickHouse/build_gcc9/../src/DataStreams/ParallelInputsProcessor.h:215
  440. ThreadFromGlobalPool::ThreadFromGlobalPool<void (DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>::*)(std::shared_ptr<DB::ThreadGroupStatus>, unsigned long), DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>*, std::shared_ptr<DB::ThreadGroupStatus>, unsigned long&>(void (DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>::*&&)(std::shared_ptr<DB::ThreadGroupStatus>, unsigned long), DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>*&&, std::shared_ptr<DB::ThreadGroupStatus>&&, unsigned long&)::{lambda()#1}::operator()() const
  441.     /usr/local/include/c++/9.1.0/bits/shared_ptr_base.h:729
  442. ThreadPoolImpl<std::thread>::worker(std::_List_iterator<std::thread>)
  443.     /usr/local/include/c++/9.1.0/bits/unique_lock.h:69
  444. execute_native_thread_routine
  445.     /home/milovidov/ClickHouse/ci/workspace/gcc/gcc-build/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/unique_ptr.h:81
  446. start_thread
  448. __clone
  451. Row 9:
  452. ───────
  453. count(): 779
  454. sym:     StackTrace::StackTrace(ucontext_t const&)
  455.     /home/milovidov/ClickHouse/build_gcc9/../src/Common/StackTrace.cpp:208
  456. DB::(anonymous namespace)::writeTraceInfo(DB::TimerType, int, siginfo_t*, void*) [clone .isra.0]
  457.     /home/milovidov/ClickHouse/build_gcc9/../src/IO/BufferBase.h:99
  460. void DB::deserializeBinarySSE2<4>(DB::PODArray<unsigned char, 4096ul, AllocatorWithHint<false, AllocatorHints::DefaultHint, 67108864ul>, 15ul, 16ul>&, DB::PODArray<unsigned long, 4096ul, AllocatorWithHint<false, AllocatorHints::DefaultHint, 67108864ul>, 15ul, 16ul>&, DB::ReadBuffer&, unsigned long)
  461.     /usr/local/lib/gcc/x86_64-pc-linux-gnu/9.1.0/include/emmintrin.h:727
  462. DB::DataTypeString::deserializeBinaryBulk(DB::IColumn&, DB::ReadBuffer&, unsigned long, double) const
  463.     /home/milovidov/ClickHouse/build_gcc9/../src/DataTypes/DataTypeString.cpp:202
  464. DB::MergeTreeReader::readData(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, DB::IDataType const&, DB::IColumn&, unsigned long, bool, unsigned long, bool)
  465.     /home/milovidov/ClickHouse/build_gcc9/../src/Storages/MergeTree/MergeTreeReader.cpp:232
  466. DB::MergeTreeReader::readRows(unsigned long, bool, unsigned long, DB::Block&)
  467.     /home/milovidov/ClickHouse/build_gcc9/../src/Storages/MergeTree/MergeTreeReader.cpp:111
  468. DB::MergeTreeRangeReader::DelayedStream::finalize(DB::Block&)
  469.     /home/milovidov/ClickHouse/build_gcc9/../src/Storages/MergeTree/MergeTreeRangeReader.cpp:35
  470. DB::MergeTreeRangeReader::startReadingChain(unsigned long, std::vector<DB::MarkRange, std::allocator<DB::MarkRange> >&)
  471.     /home/milovidov/ClickHouse/build_gcc9/../src/Storages/MergeTree/MergeTreeRangeReader.cpp:219
  472. DB::MergeTreeRangeReader::read(unsigned long, std::vector<DB::MarkRange, std::allocator<DB::MarkRange> >&)
  473.     /usr/local/include/c++/9.1.0/bits/stl_vector.h:108
  474. DB::MergeTreeRangeReader::read(unsigned long, std::vector<DB::MarkRange, std::allocator<DB::MarkRange> >&)
  475.     /usr/local/include/c++/9.1.0/bits/stl_vector.h:108
  476. DB::MergeTreeBaseSelectBlockInputStream::readFromPartImpl()
  477.     /home/milovidov/ClickHouse/build_gcc9/../src/Storages/MergeTree/MergeTreeBaseSelectBlockInputStream.cpp:158
  478. DB::MergeTreeBaseSelectBlockInputStream::readImpl()
  479.     /usr/local/include/c++/9.1.0/bits/stl_vector.h:108
  480. DB::IBlockInputStream::read()
  481.     /usr/local/include/c++/9.1.0/bits/stl_vector.h:108
  482. DB::ExpressionBlockInputStream::readImpl()
  483.     /home/milovidov/ClickHouse/build_gcc9/../src/DataStreams/ExpressionBlockInputStream.cpp:34
  484. DB::IBlockInputStream::read()
  485.     /usr/local/include/c++/9.1.0/bits/stl_vector.h:108
  486. DB::PartialSortingBlockInputStream::readImpl()
  487.     /home/milovidov/ClickHouse/build_gcc9/../src/DataStreams/PartialSortingBlockInputStream.cpp:13
  488. DB::IBlockInputStream::read()
  489.     /usr/local/include/c++/9.1.0/bits/stl_vector.h:108
  490. DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>::loop(unsigned long)
  491.     /usr/local/include/c++/9.1.0/bits/atomic_base.h:419
  492. DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>::thread(std::shared_ptr<DB::ThreadGroupStatus>, unsigned long)
  493.     /home/milovidov/ClickHouse/build_gcc9/../src/DataStreams/ParallelInputsProcessor.h:215
  494. ThreadFromGlobalPool::ThreadFromGlobalPool<void (DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>::*)(std::shared_ptr<DB::ThreadGroupStatus>, unsigned long), DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>*, std::shared_ptr<DB::ThreadGroupStatus>, unsigned long&>(void (DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>::*&&)(std::shared_ptr<DB::ThreadGroupStatus>, unsigned long), DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>*&&, std::shared_ptr<DB::ThreadGroupStatus>&&, unsigned long&)::{lambda()#1}::operator()() const
  495.     /usr/local/include/c++/9.1.0/bits/shared_ptr_base.h:729
  496. ThreadPoolImpl<std::thread>::worker(std::_List_iterator<std::thread>)
  497.     /usr/local/include/c++/9.1.0/bits/unique_lock.h:69
  498. execute_native_thread_routine
  499.     /home/milovidov/ClickHouse/ci/workspace/gcc/gcc-build/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/unique_ptr.h:81
  500. start_thread
  502. __clone
  505. Row 10:
  506. ───────
  507. count(): 666
  508. sym:     StackTrace::StackTrace(ucontext_t const&)
  509.     /home/milovidov/ClickHouse/build_gcc9/../src/Common/StackTrace.cpp:208
  510. DB::(anonymous namespace)::writeTraceInfo(DB::TimerType, int, siginfo_t*, void*) [clone .isra.0]
  511.     /home/milovidov/ClickHouse/build_gcc9/../src/IO/BufferBase.h:99
  514. void DB::deserializeBinarySSE2<4>(DB::PODArray<unsigned char, 4096ul, AllocatorWithHint<false, AllocatorHints::DefaultHint, 67108864ul>, 15ul, 16ul>&, DB::PODArray<unsigned long, 4096ul, AllocatorWithHint<false, AllocatorHints::DefaultHint, 67108864ul>, 15ul, 16ul>&, DB::ReadBuffer&, unsigned long)
  515.     /usr/local/lib/gcc/x86_64-pc-linux-gnu/9.1.0/include/emmintrin.h:727
  516. DB::DataTypeString::deserializeBinaryBulk(DB::IColumn&, DB::ReadBuffer&, unsigned long, double) const
  517.     /home/milovidov/ClickHouse/build_gcc9/../src/DataTypes/DataTypeString.cpp:202
  518. DB::MergeTreeReader::readData(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, DB::IDataType const&, DB::IColumn&, unsigned long, bool, unsigned long, bool)
  519.     /home/milovidov/ClickHouse/build_gcc9/../src/Storages/MergeTree/MergeTreeReader.cpp:232
  520. DB::MergeTreeReader::readRows(unsigned long, bool, unsigned long, DB::Block&)
  521.     /home/milovidov/ClickHouse/build_gcc9/../src/Storages/MergeTree/MergeTreeReader.cpp:111
  522. DB::MergeTreeRangeReader::DelayedStream::finalize(DB::Block&)
  523.     /home/milovidov/ClickHouse/build_gcc9/../src/Storages/MergeTree/MergeTreeRangeReader.cpp:35
  524. DB::MergeTreeRangeReader::startReadingChain(unsigned long, std::vector<DB::MarkRange, std::allocator<DB::MarkRange> >&)
  525.     /home/milovidov/ClickHouse/build_gcc9/../src/Storages/MergeTree/MergeTreeRangeReader.cpp:219
  526. DB::MergeTreeRangeReader::read(unsigned long, std::vector<DB::MarkRange, std::allocator<DB::MarkRange> >&)
  527.     /usr/local/include/c++/9.1.0/bits/stl_vector.h:108
  528. DB::MergeTreeRangeReader::read(unsigned long, std::vector<DB::MarkRange, std::allocator<DB::MarkRange> >&)
  529.     /usr/local/include/c++/9.1.0/bits/stl_vector.h:108
  530. DB::MergeTreeBaseSelectBlockInputStream::readFromPartImpl()
  531.     /home/milovidov/ClickHouse/build_gcc9/../src/Storages/MergeTree/MergeTreeBaseSelectBlockInputStream.cpp:158
  532. DB::MergeTreeBaseSelectBlockInputStream::readImpl()
  533.     /usr/local/include/c++/9.1.0/bits/stl_vector.h:108
  534. DB::IBlockInputStream::read()
  535.     /usr/local/include/c++/9.1.0/bits/stl_vector.h:108
  536. DB::ExpressionBlockInputStream::readImpl()
  537.     /home/milovidov/ClickHouse/build_gcc9/../src/DataStreams/ExpressionBlockInputStream.cpp:34
  538. DB::IBlockInputStream::read()
  539.     /usr/local/include/c++/9.1.0/bits/stl_vector.h:108
  540. DB::PartialSortingBlockInputStream::readImpl()
  541.     /home/milovidov/ClickHouse/build_gcc9/../src/DataStreams/PartialSortingBlockInputStream.cpp:13
  542. DB::IBlockInputStream::read()
  543.     /usr/local/include/c++/9.1.0/bits/stl_vector.h:108
  544. DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>::loop(unsigned long)
  545.     /usr/local/include/c++/9.1.0/bits/atomic_base.h:419
  546. DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>::thread(std::shared_ptr<DB::ThreadGroupStatus>, unsigned long)
  547.     /home/milovidov/ClickHouse/build_gcc9/../src/DataStreams/ParallelInputsProcessor.h:215
  548. ThreadFromGlobalPool::ThreadFromGlobalPool<void (DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>::*)(std::shared_ptr<DB::ThreadGroupStatus>, unsigned long), DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>*, std::shared_ptr<DB::ThreadGroupStatus>, unsigned long&>(void (DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>::*&&)(std::shared_ptr<DB::ThreadGroupStatus>, unsigned long), DB::ParallelInputsProcessor<DB::UnionBlockInputStream::Handler>*&&, std::shared_ptr<DB::ThreadGroupStatus>&&, unsigned long&)::{lambda()#1}::operator()() const
  549.     /usr/local/include/c++/9.1.0/bits/shared_ptr_base.h:729
  550. ThreadPoolImpl<std::thread>::worker(std::_List_iterator<std::thread>)
  551.     /usr/local/include/c++/9.1.0/bits/unique_lock.h:69
  552. execute_native_thread_routine
  553.     /home/milovidov/ClickHouse/ci/workspace/gcc/gcc-build/x86_64-pc-linux-gnu/libstdc++-v3/include/bits/unique_ptr.h:81
  554. start_thread
  556. __clone