How to retry Elasticsearch shard allocation

Retry Elasticsearch shard allocation that was blocked due to too many subsequent allocation failures.

Display unassigned shards.

# curl -s "http://127.0.0.1:9200/_cat/shards?v" | awk 'NR==1 {print}; $4 == "UNASSIGNED" {print}'
index            shard prirep state      docs store ip        node
books_2018-07-26 1     r      UNASSIGNED
bookmarks_index  4     r      UNASSIGNED                      

Display unassigned shards and a reason for that.

$ curl http://127.0.0.1:9200/_cluster/state | jq '.routing_table.indices | .[].shards[][] | select(.state=="UNASSIGNED") | {index: .index, shard: .shard, primary: .primary, unassigned_info: .unassigned_info}'
{
  "index": "books_2020-01-05",
  "shard": 1,
  "primary": false,
  "unassigned_info": {
    "reason": "ALLOCATION_FAILED",
    "at": "2020-01-06T09:06:38.931Z",
    "failed_attempts": 10,
    "delayed": false,
    "details": "failed to create shard, failure IOException[failed to obtain in-memory shard lock]; nested: ShardLockObtainFailedException[[books_2020-01-05][1]: obtaining shard lock timed out after 5000ms]; ",
    "allocation_status": "no_attempt"
  }
}
{
  "index": "bookmarks_index",
  "shard": 4,
  "primary": false,
  "unassigned_info": {
    "reason": "ALLOCATION_FAILED",
    "at": "2020-01-06T07:31:46.449Z",
    "failed_attempts": 5,
    "delayed": false,
    "details": "failed to create shard, failure IOException[failed to obtain in-memory shard lock]; nested: ShardLockObtainFailedException[[bookmarks_index][4]: obtaining shard lock timed out after 5000ms]; ",
    "allocation_status": "no_attempt"
  }
}

Retry Elasticsearch shard allocation that was blocked due to too many subsequent allocation failures.

$ curl -X POST http://127.0.0.1:9200/_cluster/reroute?retry_failed=true

In most simple cases you can increase the number of max entries.

$ curl --silent \
       --request PUT \
       --header 'Content-Type: application/json' \
       http://127.0.0.1:9200/bookmarks_index/_settings?pretty=true \
       --data-ascii \
'{
  "index": {
    "allocation": {
       "max_retries": 15
    }
  }                         
}'
{
  "acknowledged" : true
}

Retry operation will allocate shards that have encountered temporary issues, but this is not always the case as the reason may be elsewhere.

$ curl http://127.0.0.1:9200/_cluster/state | jq '.routing_table.indices | .[].shards[][] | select(.state=="UNASSIGNED") | {index: .index, shard: .shard, primary: .primary, unassigned_info: .unassigned_info}'
{
  "index": "books_2020-01-05",
  "shard": 1,
  "primary": false,
  "unassigned_info": {
    "reason": "ALLOCATION_FAILED",
    "at": "2020-01-06T09:06:38.931Z",
    "failed_attempts": 10,
    "delayed": false,
    "details": "failed to create shard, failure IOException[failed to obtain in-memory shard lock]; nested: ShardLockObtainFailedException[[books_2020-01-05][1]: obtaining shard lock timed out after 5000ms]; ",
    "allocation_status": "no_attempt"
  }
}

In such case get detailed explanation for shard allocations in this cluster.

$ curl http://127.0.0.1:9200/_cluster/allocation/explain?pretty=true
{
  "index": "books_2020-01-05",
  "shard": 1,
  "primary": false,
  "current_state": "unassigned",
  "unassigned_info": {
    "reason": "ALLOCATION_FAILED",
    "at": "2020-01-06T09:06:38.931Z",
    "failed_allocation_attempts": 10,
    "details": "failed to create shard, failure IOException[failed to obtain in-memory shard lock]; nested: ShardLockObtainFailedException[[books_2020-01-05][1]: obtaining shard lock timed out after 5000ms]; ",
    "last_allocation_status": "no_attempt"
  },
  "can_allocate": "no",
  "allocate_explanation": "cannot allocate because allocation is not permitted to any of the nodes",
  "node_allocation_decisions": [
    {
      "node_id": "6D0ib0mOSlueLqKDaGzlPw",
      "node_name": "elastic-node-b",
      "transport_address": "10.0.25.11:9300",
      "node_decision": "no",
      "deciders": [
        {
          "decider": "max_retry",
          "decision": "NO",
          "explanation": "shard has exceeded the maximum number of retries [10] on failed allocation attempts - manually call [/_cluster/reroute?retry_failed=true] to retry, [unassigned_info[[reason=ALLOCATION_FAILED], at[2020-01-06T09:06:38.931Z], failed_attempts[10], delayed=false, details[failed to create shard, failure IOException[failed to obtain in-memory shard lock]; nested: ShardLockObtainFailedException[[books_2020-01-05][1]: obtaining shard lock timed out after 5000ms]; ], allocation_status[no_attempt]]]"
        },
        {
          "decider": "filter",
          "decision": "NO",
          "explanation": "node matches cluster setting [cluster.routing.allocation.exclude] filters [_ip:\"10.0.25.11 OR 10.0.25.12\"]"
        }
      ]
    },
    {
      "node_id": "PMmTS-8WTLCd5S69DwUVTg",
      "node_name": "elastic-node-c",
      "transport_address": "10.0.25.12:9300",
      "node_decision": "no",
      "deciders": [
        {
          "decider": "max_retry",
          "decision": "NO",
          "explanation": "shard has exceeded the maximum number of retries [10] on failed allocation attempts - manually call [/_cluster/reroute?retry_failed=true] to retry, [unassigned_info[[reason=ALLOCATION_FAILED], at[2020-01-06T09:06:38.931Z], failed_attempts[10], delayed=false, details[failed to create shard, failure IOException[failed to obtain in-memory shard lock]; nested: ShardLockObtainFailedException[[books_2020-01-05][1]: obtaining shard lock timed out after 5000ms]; ], allocation_status[no_attempt]]]"
        },
        {
          "decider": "filter",
          "decision": "NO",
          "explanation": "node matches cluster setting [cluster.routing.allocation.exclude] filters [_ip:\"10.0.25.11 OR 10.0.25.12\"]"
        }
      ]
    }
  ]
}

In this specific case exclude filter prevents this shard from allocation, so you have to take the appropriate steps to handle this situation.