Search backpressure

Search backpressure is a mechanism used to identify resource-intensive search requests and cancel them when the node is under duress. If a search request on a node or shard has breached the resource limits and does not recover within a certain threshold, it is rejected. These thresholds are dynamic and configurable through cluster settings using the /_cluster/settings API endpoint.

Measuring resource consumption

To decide whether to apply search backpressure, OpenSearch periodically measures the following resource consumption statistics for each search request:

• CPU usage
• Heap usage
• Elapsed time

An observer thread periodically measures the resource usage of the node. If OpenSearch determines that the node is under duress, OpenSearch examines the resource usage of each search task and search shard task and compares it against configurable thresholds. OpenSearch considers CPU usage, heap usage, and elapsed time and assigns each task a cancellation score that is then used to cancel the most resource-intensive tasks.

OpenSearch limits the number of cancellations to a fraction of successful task completions. Additionally, it limits the number of cancellations per unit time. OpenSearch continues to monitor and cancel tasks until the node is no longer under duress.

Canceled queries

If a query is canceled, OpenSearch may return partial results if some shards failed. If all shards failed, OpenSearch returns an error from the server similar to the following error:

{
  "error": {
    "root_cause": [
      {
          "type": "task_cancelled_exception",
          "reason": "cancelled task with reason: cpu usage exceeded [17.9ms >= 15ms], elapsed time exceeded [1.1s >= 300ms]"
      },
      {
          "type": "task_cancelled_exception",
          "reason": "cancelled task with reason: elapsed time exceeded [1.1s >= 300ms]"
      }
    ],
    "type": "search_phase_execution_exception",
    "reason": "all shards failed",
    "phase": "query",
    "grouped": true,
    "failed_shards": [
      {
        "shard": 0,
        "index": "foobar",
        "node": "7yIqOeMfRyWW1rHs2S4byw",
        "reason": {
            "type": "task_cancelled_exception",
            "reason": "cancelled task with reason: cpu usage exceeded [17.9ms >= 15ms], elapsed time exceeded [1.1s >= 300ms]"
        }
      },
      {
        "shard": 1,
        "index": "foobar",
        "node": "7yIqOeMfRyWW1rHs2S4byw",
        "reason": {
            "type": "task_cancelled_exception",
            "reason": "cancelled task with reason: elapsed time exceeded [1.1s >= 300ms]"
        }
      }
    ]
  },
  "status": 500
}

Search backpressure modes

Search backpressure runs in monitor_only (default), enforced, or disabled mode. In the enforced mode, the server rejects search requests. In the monitor_only mode, the server does not actually cancel search requests but tracks statistics about them. You can specify the mode in the search_backpressure.mode parameter.

Search backpressure settings

Search backpressure adds several settings to the standard OpenSearch cluster settings. These settings are dynamic, so you can change the default behavior of this feature without restarting your cluster.

To configure these settings, send a PUT request to /_cluster/settings:

PUT /_cluster/settings
{
  "persistent": {
    "search_backpressure": {
      "mode": "monitor_only"
    }
  }
}

copy

Search Backpressure Stats API

Introduced 2.4

You can use the nodes stats API operation to monitor server-side request cancellations.

Example request

To retrieve the statistics, use the following request:

GET _nodes/stats/search_backpressure

Example response

The response contains server-side request cancellation statistics:

{
  "_nodes": {
    "total": 1,
    "successful": 1,
    "failed": 0
  },
  "cluster_name": "runTask",
  "nodes": {
    "T7aqO6zaQX-lt8XBWBYLsA": {
      "timestamp": 1667409521070,
      "name": "runTask-0",
      "transport_address": "127.0.0.1:9300",
      "host": "127.0.0.1",
      "ip": "127.0.0.1:9300",
      "roles": [
      ],
      "attributes": {
        "testattr": "test",
        "shard_indexing_pressure_enabled": "true"
      },
      "search_backpressure": {
        "search_task": {
          "resource_tracker_stats": {
            "heap_usage_tracker": {
              "cancellation_count": 57,
              "current_max_bytes": 5739204,
              "current_avg_bytes": 962465,
              "rolling_avg_bytes": 4009239
            },
            "elapsed_time_tracker": {
              "cancellation_count": 97,
              "current_max_millis": 15902,
              "current_avg_millis": 9705
            },
            "cpu_usage_tracker": {
              "cancellation_count": 64,
              "current_max_millis": 8483,
              "current_avg_millis": 7843
            }
          },
          "cancellation_stats": {
            "cancellation_count": 102,
            "cancellation_limit_reached_count": 25
          }
        },
        "search_shard_task": {
          "resource_tracker_stats": {
            "heap_usage_tracker": {
              "cancellation_count": 34,
              "current_max_bytes": 1203272,
              "current_avg_bytes": 700267,
              "rolling_avg_bytes": 1156270
            },
            "cpu_usage_tracker": {
              "cancellation_count": 318,
              "current_max_millis": 731,
              "current_avg_millis": 303
            },
            "elapsed_time_tracker": {
              "cancellation_count": 310,
              "current_max_millis": 1305,
              "current_avg_millis": 649
            }
          },
          "cancellation_stats": {
            "cancellation_count": 318,
            "cancellation_limit_reached_count": 97
          }
        },
        "mode": "enforced"
      }
    }
  }
}

Response fields

The response contains the following fields.

resource_tracker_stats

The resource_tracker_stats object contains the statistics for each resource tracker: elapsed_time_tracker, heap_usage_tracker, and cpu_usage_tracker.

elapsed_time_tracker

The elapsed_time_tracker object contains the following statistics related to the elapsed time.

heap_usage_tracker

The heap_usage_tracker object contains the following statistics related to the heap usage.

cpu_usage_tracker

The cpu_usage_tracker object contains the following statistics related to the CPU usage.

cancellation_stats

The cancellation_stats object contains the following statistics for the tasks that are marked for cancellation.