Top N queries

Monitoring the top N queries in query insights features can help you gain real-time insights into the top queries with high latency within a certain time frame (for example, the last hour).

Configuring top N query monitoring

You can configure top N query monitoring by the following metric types:

  • latency
  • cpu
  • memory

Each metric has a set of corresponding settings:

For example, to enable top N query monitoring by CPU usage, set search.insights.top_queries.cpu.enabled to true. For more information about ways to specify dynamic settings, see Dynamic settings.

It’s important to exercise caution when enabling this feature because it can consume system resources.

Enabling top N query monitoring

When you install the query-insights plugin, top N query monitoring is disabled by default. To enable top N query monitoring, update the dynamic settings for the desired metric types. These settings enable the corresponding collectors and aggregators in the running cluster. For example, to enable top N query monitoring by latency, update the search.insights.top_queries.latency.enabled setting:

  1. PUT _cluster/settings
  2. {
  3.   "persistent" : {
  4.     "search.insights.top_queries.latency.enabled" : true
  5.   }
  6. }

copy

Configuring the window size

To configure the monitoring window size, update the window_size setting for the desired metric type. For example, to collect the top N queries by latency in a 60-minute window, update the search.insights.top_queries.latency.window_size setting:

  1. PUT _cluster/settings
  2. {
  3.   "persistent" : {
  4.     "search.insights.top_queries.latency.window_size" : "60m"
  5.   }
  6. }

copy

Configuring the value of N

To configure the value of N, update the top_n_size setting for the desired metric type. For example, to collect the top 10 queries by latency, update the insights.top_queries.latency.top_n_size setting:

  1. PUT _cluster/settings
  2. {
  3.   "persistent" : {
  4.     "search.insights.top_queries.latency.top_n_size" : 10
  5.   }
  6. }

copy

Monitoring the top N queries

You can use the Insights API endpoint to obtain the top N queries for all metric types:

  1. GET /_insights/top_queries

copy

Specify a metric type to filter the response:

  1. GET /_insights/top_queries?type=latency

copy

  1. GET /_insights/top_queries?type=cpu

copy

  1. GET /_insights/top_queries?type=memory

copy

Exporting top N query data

You can configure your desired exporter to export top N query data to different sinks, allowing for better monitoring and analysis of your OpenSearch queries. Currently, the following exporters are supported:

Configuring a debug exporter

To configure a debug exporter, update the exporter setting for the desired metric type. For example, to export the top N queries by latency using the debug exporter, send the following request:

  1. PUT _cluster/settings
  2. {
  3.   "persistent" : {
  4.      "search.insights.top_queries.latency.exporter.type" : "debug"
  5.   }
  6. }

copy

Configuring a local index exporter

A local index exporter allows you to export the top N queries to local OpenSearch indexes. The default index pattern for top N query indexes is top_queries-YYYY.MM.dd. All top queries from the same day are saved to the same index, and a new index is created each day. You can change the default index pattern to use other date formats. For more information about supported formats, see DateTimeFormat.

To configure the local index exporter for the top N queries by latency, send the following request:

  1. PUT _cluster/settings
  2. {
  3.   "persistent" : {
  4.     "search.insights.top_queries.latency.exporter.type" : "local_index",
  5.     "search.insights.top_queries.latency.exporter.config.index" : "YYYY.MM.dd"
  6.   }
  7. }

copy