Nodes hot threads

Nodes hot threads

The nodes hot threads endpoint provides information about busy JVM threads for selected cluster nodes. It provides a unique view of the of activity each node.

Example

GET /_nodes/hot_threads

Path and HTTP methods

GET /_nodes/hot_threads
GET /_nodes/{nodeId}/hot_threads

URL parameters

You can include the following URL parameters in your request. All parameters are optional.

Parameter	Type	Description
nodeId	String	A comma-separated list of node IDs to filter results. Supports node filters. Defaults to `_all`.
snapshots	Integer	Number of samples of thread stacktraces. Defaults to `10`.
interval	TimeValue	Interval between consecutive samples. Defaults to `500ms`.
threads	Integer	A number of top busiest threads to return information about. Defaults to `3`.
ignore_idle_threads	Boolean	Don’t show threads that are in known-idle states, such as waiting on a socket select or pulling from an empty task queue. Defaults to `true`.
type	String	Supported thread types are `cpu`, `wait`, or `block`. Defaults to `cpu`.
timeout	TimeValue	A request timeout. Defaults to `30s`.

Response

Unlike the majority of OpenSearch API responses, this response is in a text format.

It consists of one section per each cluster node included in the response.

Each section starts with a single line containing the following segments:

Line segment	Description
`:::`	Line start (a distinct visual symbol).
`{global-eu-35}`	Node name.
`{uFPbKLDOTlOmdnwUlKW8sw}`	NodeId.
`{OAM8OT5CQAyasWuIDeVyUA}`	EphemeralId.
`{global-eu-35.local}`	Host name.
`{[gdv2:a284:2acv:5fa6:0:3a2:7260:74cf]:9300}`	Host address.
`{dimr}`	Node roles (d=data, i=ingest, m=cluster manager, r=remote cluster client).
`{zone=west-a2, shard_indexing_pressure_enabled=true}`	Node attributes.

Then follows information about threads of selected type.

::: {global-eu-35}{uFPbKLDOTlOmdnwUlKW8sw}{OAM8OT5CQAyasWuIDeVyUA}{global-eu-35.local}{[gdv2:a284:2acv:5fa6:0:3a2:7260:74cf]:9300}{dimr}{zone=west-a2, shard_indexing_pressure_enabled=true}
   Hot threads at 2022-04-01T15:15:27.658Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
    0.1% (645micros out of 500ms) cpu usage by thread 'opensearch[global-eu-35][transport_worker][T#7]'
     4/10 snapshots sharing following 3 elements
       io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
       io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
       java.base@11.0.14.1/java.lang.Thread.run(Thread.java:829)
::: {global-eu-62}{4knOxAdERlOB19zLQIT1bQ}{HJuZs2HiQ_-8Elj0Fvi_1g}{global-eu-62.local}{[gdv2:a284:2acv:5fa6:0:3a2:bba6:fe3f]:9300}{dimr}{zone=west-a2, shard_indexing_pressure_enabled=true}
   Hot threads at 2022-04-01T15:15:27.659Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
   18.7% (93.4ms out of 500ms) cpu usage by thread 'opensearch[global-eu-62][transport_worker][T#3]'
     6/10 snapshots sharing following 3 elements
       io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
       io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
       java.base@11.0.14.1/java.lang.Thread.run(Thread.java:829)
::: {global-eu-44}{8WW3hrkcTwGvgah_L8D_jw}{Sok7spHISFyol0jFV6i0kw}{global-eu-44.local}{[gdv2:a284:2acv:5fa6:0:3a2:9120:e79e]:9300}{dimr}{zone=west-a2, shard_indexing_pressure_enabled=true}
   Hot threads at 2022-04-01T15:15:27.659Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
   42.6% (212.7ms out of 500ms) cpu usage by thread 'opensearch[global-eu-44][write][T#5]'
     2/10 snapshots sharing following 43 elements
       java.base@11.0.14.1/sun.nio.ch.IOUtil.write1(Native Method)
       java.base@11.0.14.1/sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:254)
       io.netty.channel.nio.NioEventLoop.wakeup(NioEventLoop.java:787)
       io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:846)
       io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:815)
       io.netty.channel.AbstractChannelHandlerContext.safeExecute(AbstractChannelHandlerContext.java:989)
       io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:796)
       io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:758)
       io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1020)
       io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:311)
       org.opensearch.transport.netty4.Netty4TcpChannel.sendMessage(Netty4TcpChannel.java:159)
       app//org.opensearch.transport.OutboundHan...

Required permissions

If you use the security plugin, make sure you set the following permissions: cluster:monitor/nodes/hot_threads.