Total number of shards for an index on a single node exceeded

Total number of shards for an index on a single node exceeded

Elasticsearch tries to take advantage of all the available resources by distributing data (index shards) among nodes in the cluster.

Users might want to influence this data distribution by configuring the index.routing.allocation.total_shards_per_node index setting to a custom value (for e.g. 1 in case of a highly trafficked index). Various configurations limiting how many shards an index can have located on one node can lead to shards being unassigned due to the cluster not having enough nodes to satisfy the index configuration.

In order to fix this follow the next steps:

Elasticsearch Service Self-managed

In order to get the shards assigned we’ll need to increase the number of shards that can be collocated on a node. We’ll achieve this by inspecting the configuration for the index.routing.allocation.total_shards_per_node index setting and increasing the configured value for the indices that have shards unassigned.

Use Kibana

  1. Log in to the Elastic Cloud console.

  2. On the Elasticsearch Service panel, click the name of your deployment.

    If the name of your deployment is disabled your Kibana instances might be unhealthy, in which case please contact Elastic Support. If your deployment doesn’t include Kibana, all you need to do is enable it first.

  3. Open your deployment’s side navigation menu (placed under the Elastic logo in the upper left corner) and go to Dev Tools > Console.

    Kibana Console

  4. Inspect the index.routing.allocation.total_shards_per_node index setting for the index with unassigned shards:

    1. resp = client.indices.get_settings(
    2.     index="my-index-000001",
    3.     name="index.routing.allocation.total_shards_per_node",
    4.     flat_settings=True,
    5. )
    6. print(resp)
    1. response = client.indices.get_settings(
    2.   index: 'my-index-000001',
    3.   name: 'index.routing.allocation.total_shards_per_node',
    4.   flat_settings: true
    5. )
    6. puts response
    1. const response = await client.indices.getSettings({
    2.   index: "my-index-000001",
    3.   name: "index.routing.allocation.total_shards_per_node",
    4.   flat_settings: "true",
    5. });
    6. console.log(response);
    1. GET /my-index-000001/_settings/index.routing.allocation.total_shards_per_node?flat_settings

    The response will look like this:

    1. {
    2.   "my-index-000001": {
    3.     "settings": {
    4.       "index.routing.allocation.total_shards_per_node": "1" 
    5.     }
    6.   }
    7. }

    Represents the current configured value for the total number of shards that can reside on one node for the my-index-000001 index.

  5. Increase the value for the total number of shards that can be assigned on one node to a higher value:

    1. resp = client.indices.put_settings(
    2.     index="my-index-000001",
    3.     settings={
    4.         "index": {
    5.             "routing.allocation.total_shards_per_node": "2"
    6.         }
    7.     },
    8. )
    9. print(resp)
    1. response = client.indices.put_settings(
    2.   index: 'my-index-000001',
    3.   body: {
    4.     index: {
    5.       'routing.allocation.total_shards_per_node' => '2'
    6.     }
    7.   }
    8. )
    9. puts response
    1. const response = await client.indices.putSettings({
    2.   index: "my-index-000001",
    3.   settings: {
    4.     index: {
    5.       "routing.allocation.total_shards_per_node": "2",
    6.     },
    7.   },
    8. });
    9. console.log(response);
    1. PUT /my-index-000001/_settings
    2. {
    3.   "index" : {
    4.     "routing.allocation.total_shards_per_node" : "2" 
    5.   }
    6. }

    The new value for the total_shards_per_node configuration for the my-index-000001 index is increased from the previous value of 1 to 2. The total_shards_per_node configuration can also be set to -1, which represents no upper bound with regards to how many shards of the same index can reside on one node.

In order to get the shards assigned you can add more nodes to your Elasticsearch cluster and assing the index’s target tier node role to the new nodes.

To inspect which tier is an index targeting for assignment, use the get index setting API to retrieve the configured value for the index.routing.allocation.include._tier_preference setting:

  1. resp = client.indices.get_settings(
  2.     index="my-index-000001",
  3.     name="index.routing.allocation.include._tier_preference",
  4.     flat_settings=True,
  5. )
  6. print(resp)
  1. response = client.indices.get_settings(
  2.   index: 'my-index-000001',
  3.   name: 'index.routing.allocation.include._tier_preference',
  4.   flat_settings: true
  5. )
  6. puts response
  1. const response = await client.indices.getSettings({
  2.   index: "my-index-000001",
  3.   name: "index.routing.allocation.include._tier_preference",
  4.   flat_settings: "true",
  5. });
  6. console.log(response);
  1. GET /my-index-000001/_settings/index.routing.allocation.include._tier_preference?flat_settings

The response will look like this:

  1. {
  2.   "my-index-000001": {
  3.     "settings": {
  4.       "index.routing.allocation.include._tier_preference": "data_warm,data_hot" 
  5.     }
  6.   }
  7. }

Represents a comma separated list of data tier node roles this index is allowed to be allocated on, the first one in the list being the one with the higher priority i.e. the tier the index is targeting. e.g. in this example the tier preference is data_warm,data_hot so the index is targeting the warm tier and more nodes with the data_warm role are needed in the Elasticsearch cluster.

Alternatively, if adding more nodes to the Elasticsearch cluster is not desired, inspecting the configuration for the index.routing.allocation.total_shards_per_node index setting and increasing the configured value will allow more shards to be assigned on the same node.

  1. Inspect the index.routing.allocation.total_shards_per_node index setting for the index with unassigned shards:

    1. resp = client.indices.get_settings(
    2.     index="my-index-000001",
    3.     name="index.routing.allocation.total_shards_per_node",
    4.     flat_settings=True,
    5. )
    6. print(resp)
    1. response = client.indices.get_settings(
    2.   index: 'my-index-000001',
    3.   name: 'index.routing.allocation.total_shards_per_node',
    4.   flat_settings: true
    5. )
    6. puts response
    1. const response = await client.indices.getSettings({
    2.   index: "my-index-000001",
    3.   name: "index.routing.allocation.total_shards_per_node",
    4.   flat_settings: "true",
    5. });
    6. console.log(response);
    1. GET /my-index-000001/_settings/index.routing.allocation.total_shards_per_node?flat_settings

    The response will look like this:

    1. {
    2.   "my-index-000001": {
    3.     "settings": {
    4.       "index.routing.allocation.total_shards_per_node": "1" 
    5.     }
    6.   }
    7. }

    Represents the current configured value for the total number of shards that can reside on one node for the my-index-000001 index.

  2. Increase the total number of shards that can be assigned on one node or reset the value to unbounded (-1):

    1. resp = client.indices.put_settings(
    2.     index="my-index-000001",
    3.     settings={
    4.         "index": {
    5.             "routing.allocation.total_shards_per_node": -1
    6.         }
    7.     },
    8. )
    9. print(resp)
    1. response = client.indices.put_settings(
    2.   index: 'my-index-000001',
    3.   body: {
    4.     index: {
    5.       'routing.allocation.total_shards_per_node' => -1
    6.     }
    7.   }
    8. )
    9. puts response
    1. const response = await client.indices.putSettings({
    2.   index: "my-index-000001",
    3.   settings: {
    4.     index: {
    5.       "routing.allocation.total_shards_per_node": -1,
    6.     },
    7.   },
    8. });
    9. console.log(response);
    1. PUT /my-index-000001/_settings
    2. {
    3.   "index" : {
    4.     "routing.allocation.total_shards_per_node" : -1
    5.   }
    6. }