Nodes hot threads

Introduced 1.0

The nodes hot threads endpoint provides information about busy JVM threads for selected cluster nodes. It provides a unique view of the of activity each node.

Example

  1. GET /_nodes/hot_threads

copy

Path and HTTP methods

  1. GET /_nodes/hot_threads
  2. GET /_nodes/<nodeId>/hot_threads

Path parameters

You can include the following optional path parameter in your request.

ParameterTypeDescription
nodeIdStringA comma-separated list of node IDs used to filter results. Supports node filters. Defaults to _all.

Query parameters

You can include the following query parameters in your request. All query parameters are optional.

ParameterTypeDescription
snapshotsIntegerThe number of samples of thread stacktraces. Defaults to 10.
intervalTimeThe interval between consecutive samples. Defaults to 500ms.
threadsIntegerThe number of the busiest threads to return information about. Defaults to 3.
ignore_idle_threadsBooleanDon’t show threads that are in known idle states, such as waiting on a socket select or pulling from an empty task queue. Defaults to true.
typeStringSupported thread types are cpu, wait, or block. Defaults to cpu.
timeoutTimeSets the time limit for node response. Default value is 30s.

Example request

  1. GET /_nodes/hot_threads

copy

Example response

  1. ::: {opensearch}{F-ByTQzVQ3GQeYzQJArJGQ}{GxbcLdCATPWggOuQHJAoCw}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true}
  2.    Hot threads at 2022-09-29T19:46:44.533Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
  4.     0.1% (455.5micros out of 500ms) cpu usage by thread 'ScheduledMetricCollectorsExecutor'
  5.      10/10 snapshots sharing following 2 elements
  6.        java.base@17.0.4/java.lang.Thread.sleep(Native Method)
  7.        org.opensearch.performanceanalyzer.collectors.ScheduledMetricCollectorsExecutor.run(ScheduledMetricCollectorsExecutor.java:100)

Response

Unlike the majority of OpenSearch API responses, this response is in a text format.

It consists of one section per each cluster node included in the response.

Each section starts with a single line containing the following segments:

Line segmentDescription
::: Line start (a distinct visual symbol).
{global-eu-35}Node name.
{uFPbKLDOTlOmdnwUlKW8sw}NodeId.
{OAM8OT5CQAyasWuIDeVyUA}EphemeralId.
{global-eu-35.local}Host name.
{[gdv2:a284:2acv:5fa6:0:3a2:7260:74cf]:9300}Host address.
{dimr}Node roles (d=data, i=ingest, m=cluster manager, r=remote cluster client).
{zone=west-a2, shard_indexing_pressure_enabled=true}Node attributes.

Then information about threads of the selected type is provided.

  1. ::: {global-eu-35}{uFPbKLDOTlOmdnwUlKW8sw}{OAM8OT5CQAyasWuIDeVyUA}{global-eu-35.local}{[gdv2:a284:2acv:5fa6:0:3a2:7260:74cf]:9300}{dimr}{zone=west-a2, shard_indexing_pressure_enabled=true}
  2.    Hot threads at 2022-04-01T15:15:27.658Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
  4.     0.1% (645micros out of 500ms) cpu usage by thread 'opensearch[global-eu-35][transport_worker][T#7]'
  5.      4/10 snapshots sharing following 3 elements
  6.        io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
  7.        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
  8.        java.base@11.0.14.1/java.lang.Thread.run(Thread.java:829)
  9. ::: {global-eu-62}{4knOxAdERlOB19zLQIT1bQ}{HJuZs2HiQ_-8Elj0Fvi_1g}{global-eu-62.local}{[gdv2:a284:2acv:5fa6:0:3a2:bba6:fe3f]:9300}{dimr}{zone=west-a2, shard_indexing_pressure_enabled=true}
  10.    Hot threads at 2022-04-01T15:15:27.659Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
  12.    18.7% (93.4ms out of 500ms) cpu usage by thread 'opensearch[global-eu-62][transport_worker][T#3]'
  13.      6/10 snapshots sharing following 3 elements
  14.        io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
  15.        io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
  16.        java.base@11.0.14.1/java.lang.Thread.run(Thread.java:829)
  17. ::: {global-eu-44}{8WW3hrkcTwGvgah_L8D_jw}{Sok7spHISFyol0jFV6i0kw}{global-eu-44.local}{[gdv2:a284:2acv:5fa6:0:3a2:9120:e79e]:9300}{dimr}{zone=west-a2, shard_indexing_pressure_enabled=true}
  18.    Hot threads at 2022-04-01T15:15:27.659Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
  20.    42.6% (212.7ms out of 500ms) cpu usage by thread 'opensearch[global-eu-44][write][T#5]'
  21.      2/10 snapshots sharing following 43 elements
  22.        java.base@11.0.14.1/sun.nio.ch.IOUtil.write1(Native Method)
  23.        java.base@11.0.14.1/sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:254)
  24.        io.netty.channel.nio.NioEventLoop.wakeup(NioEventLoop.java:787)
  25.        io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:846)
  26.        io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:815)
  27.        io.netty.channel.AbstractChannelHandlerContext.safeExecute(AbstractChannelHandlerContext.java:989)
  28.        io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:796)
  29.        io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:758)
  30.        io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1020)
  31.        io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:311)
  32.        org.opensearch.transport.netty4.Netty4TcpChannel.sendMessage(Netty4TcpChannel.java:159)
  33.        app//org.opensearch.transport.OutboundHan...

Required permissions

If you use the Security plugin, make sure you set the following permissions: cluster:monitor/nodes/hot_threads.