Top N queries

Monitoring the top N queries using query insights allows you to gain real-time visibility into the queries with the highest latency or resource consumption in a specified time period (for example, the last hour).

Configuring top N query monitoring

You can configure top N query monitoring by the following metric types:

  • latency
  • cpu
  • memory

Each metric has a set of corresponding settings:

For example, to enable top N query monitoring by CPU usage, set search.insights.top_queries.cpu.enabled to true. For more information about ways to specify dynamic settings, see Dynamic settings.

It’s important to exercise caution when enabling this feature because it can consume system resources.

Enabling top N query monitoring

When you install the query-insights plugin, top N query monitoring is disabled by default. To enable top N query monitoring, update the dynamic settings for the desired metric types. These settings enable the corresponding collectors and aggregators in the running cluster. For example, to enable top N query monitoring by latency, update the search.insights.top_queries.latency.enabled setting:

  1. PUT _cluster/settings
  2. {
  3.   "persistent" : {
  4.     "search.insights.top_queries.latency.enabled" : true
  5.   }
  6. }

copy

Configuring the window size

To configure the monitoring window size, update the window_size setting for the desired metric type. For example, to collect the top N queries by latency in a 60-minute window, update the search.insights.top_queries.latency.window_size setting:

  1. PUT _cluster/settings
  2. {
  3.   "persistent" : {
  4.     "search.insights.top_queries.latency.window_size" : "60m"
  5.   }
  6. }

copy

Configuring the value of N

To configure the value of N, update the top_n_size setting for the desired metric type. For example, to collect the top 10 queries by latency, update the insights.top_queries.latency.top_n_size setting:

  1. PUT _cluster/settings
  2. {
  3.   "persistent" : {
  4.     "search.insights.top_queries.latency.top_n_size" : 10
  5.   }
  6. }

copy

Monitoring the top N queries

You can use the Insights API endpoint to retrieve the top N queries. This API returns top N latency results by default.

  1. GET /_insights/top_queries

copy

Specify the type parameter to retrieve the top N results for other metric types. The results will be sorted in descending order based on the specified metric type.

  1. GET /_insights/top_queries?type=latency

copy

  1. GET /_insights/top_queries?type=cpu

copy

  1. GET /_insights/top_queries?type=memory

copy

To specify a time range for querying top N results, use the from and to parameters in ISO8601 format: YYYY-MM-DD'T'HH:mm:ss.SSSZ. For example, to retrieve the top N queries from August 25, 2024, at 15:00 UTC to August 30, 2024, at 17:00 UTC, send the following request:

  1. GET /_insights/top_queries?from=2024-08-25T15:00:00.000Z&to=2024-08-30T17:00:00.000Z

copy

If you have a local index exporter enabled, historical queries stored in local OpenSearch indexes will also be included in the specified time range.

If your query returns no results, ensure that top N query monitoring is enabled for the target metric type and that search requests were made within the current time window.

Exporting top N query data

You can configure your desired exporter to export top N query data to different sinks, allowing for better monitoring and analysis of your OpenSearch queries. Currently, the following exporters are supported:

Configuring a debug exporter

To configure a debug exporter, update the exporter setting for the desired metric type. For example, to export the top N queries by latency using the debug exporter, send the following request:

  1. PUT _cluster/settings
  2. {
  3.   "persistent" : {
  4.      "search.insights.top_queries.latency.exporter.type" : "debug"
  5.   }
  6. }

copy

Configuring a local index exporter

A local index exporter allows you to export the top N queries to local OpenSearch indexes. The default index pattern for top N query indexes is top_queries-YYYY.MM.dd. All top queries from the same day are saved to the same index, and a new index is created each day. You can change the default index pattern to use other date formats. For more information about supported formats, see DateTimeFormat.

To configure the local index exporter for the top N queries by latency, send the following request:

  1. PUT _cluster/settings
  2. {
  3.   "persistent" : {
  4.     "search.insights.top_queries.latency.exporter.type" : "local_index",
  5.     "search.insights.top_queries.latency.exporter.config.index" : "YYYY.MM.dd"
  6.   }
  7. }

copy