Recovering from an Index Node Failure

When an Elasticsearch or OpenSearch node backing a JanusGraph mixed index dies, the storage tier keeps committing while the search tier drops shards, and the graph starts answering the same query two different ways depending on which shards survived. This procedure sits under Reindexing and Recovery and narrows it to one incident: detecting the loss, assessing cluster health, restoring capacity, and reconciling the index so it once again matches storage. The specific failure it prevents is silent under-reporting after a partial recovery — a search cluster that returns to green on its own but is missing every document that should have been written during the outage, so traversals look healthy while omitting durable vertices. The recovery is ordered deliberately: restore the search cluster first, then reconcile the graph, never the reverse.

Index node failure recovery flow from detect through assess and restore to reconcile Recovery proceeds in four ordered stages. Detect: a node or shard loss surfaces as a red or yellow cluster health and unassigned shards. Assess: read _cat/health and _cluster/allocation/explain to determine whether replicas can recover the data or the node must be replaced. Restore: bring back replicas or replace the failed node so shards reassign and health returns to green. Reconcile: run a targeted JanusGraph reindex bounded to the outage window to replay mutations the index missed, then verify storage-versus-index parity. 1 · Detect red/yellow · unassigned shards 2 · Assess allocation/explain 3 · Restore replicas / replace node → green 4 · Reconcile targeted reindex + parity search tier first graph last
Restore the search cluster to green before reconciling the graph; a targeted reindex of the outage window replays the mutations the index missed.

Prerequisites

Confirm these before acting. The recovery order — search tier first, then graph — matters, and reconciling before the search cluster is green just reindexes into a still-degraded cluster.

  • A JanusGraph deployment on JanusGraph 0.6.x or 1.0.x with an Elasticsearch (7.x/8.x) or OpenSearch (1.x/2.x) index backend and a CQL storage tier that is itself healthy — this is an index-tier incident, not a storage one.
  • _cat and _cluster API access to the index backend, plus permission to reroute shards and, if needed, provision a replacement node.
  • A known outage window. The approximate start time of the node loss, so the reconciling reindex can be bounded rather than a full rebuild. Pull it from the index backend logs or your monitoring.
  • Replica configuration in view. Know the number_of_replicas for the JanusGraph index; a zero-replica index cannot self-heal from a node loss and must be reindexed from storage.
  • A storage-backed read fallback for the affected predicate while the index is degraded, the same fallback used in Running a Zero-Downtime Reindex.

Step 1 — Detect the loss and read cluster health

Start at the top-level health. red means at least one primary shard is unassigned and some data is unqueryable; yellow means primaries are assigned but replicas are not, so the data is present but has no redundancy.

bash 
# Overall health and per-index breakdown.
curl -s "http://10.0.2.20:9200/_cat/health?v"
curl -s "http://10.0.2.20:9200/_cat/indices/janusgraph_*?v&health=red,yellow"
curl -s "http://10.0.2.20:9200/_cat/nodes?v"   # is a node actually gone?

Verify: confirm the status and how many shards are unassigned.

bash 
curl -s "http://10.0.2.20:9200/_cluster/health?level=indices" \
  | jq '{status, unassigned_shards, active_shards_percent_as_number}'

Step 2 — Assess why shards are unassigned

Do not guess the cause. allocation/explain tells you whether a shard can recover from an existing replica, is waiting on a returning node, or has no surviving copy and must be rebuilt.

bash 
# Explain the first unassigned shard: node left, allocation throttled, or no copy?
curl -s -XGET "http://10.0.2.20:9200/_cluster/allocation/explain" \
  -H 'Content-Type: application/json' -d '{
    "index": "janusgraph_byStatusMixed",
    "shard": 0,
    "primary": true
  }' | jq '{current_state, unassigned_info, can_allocate}'

Interpret the result: NODE_LEFT with a recoverable replica means the search cluster will self-heal once the node returns or a replica promotes; no_valid_shard_copy means the only copy died with the node and the data must come back from storage via reindex.

Verify: decide the branch explicitly.

bash 
# If any primary reports no_valid_shard_copy, this is a data-loss recovery,
# not a reassignment. Reindex from storage is mandatory for those shards.
curl -s "http://10.0.2.20:9200/_cluster/allocation/explain" \
  -H 'Content-Type: application/json' -d '{"index":"janusgraph_byStatusMixed","shard":0,"primary":true}' \
  | jq -r '.can_allocate'

Step 3 — Restore replicas or replace the node

If a replica survives, let the search cluster reassign it — or force it if allocation is stuck. If the node is gone for good, replace it and let shards rebalance onto it.

bash 
# Case A — replica exists but allocation is stuck: nudge a retry.
curl -s -XPOST "http://10.0.2.20:9200/_cluster/reroute?retry_failed=true"

# Case B — zero-replica index that lost a primary: add a replica so future
# losses self-heal, then the data must still be reindexed from storage.
curl -s -XPUT "http://10.0.2.20:9200/janusgraph_byStatusMixed/_settings" \
  -H 'Content-Type: application/json' -d '{"index":{"number_of_replicas":1}}'

# Case C — replacement node joined: confirm it is picking up shards.
curl -s "http://10.0.2.20:9200/_cat/recovery/janusgraph_*?v&active_only=true"

Verify: health must climb back to green (or at least yellow with all primaries assigned) before you touch JanusGraph.

bash 
curl -s "http://10.0.2.20:9200/_cluster/health/janusgraph_*" \
  | jq '{status, unassigned_shards}'   # want status green, unassigned_shards 0

Step 4 — Reconcile the graph with a targeted reindex

The search cluster is healthy, but any mutation JanusGraph dispatched during the outage that hit a dead shard was lost — the storage row is durable, the index document is not. Replay only the outage window rather than rebuilding the whole index. Register first if the index dropped out of ENABLED, then reindex.

java 
// If the index reverted below ENABLED during the incident, re-register first.
JanusGraphManagement mgmt = graph.openManagement();
mgmt.updateIndex(mgmt.getGraphIndex("byStatusMixed"), SchemaAction.REGISTER_INDEX);
mgmt.commit();
ManagementSystem.awaitGraphIndexStatus(graph, "byStatusMixed")
    .status(SchemaStatus.REGISTERED)
    .timeout(10, java.time.temporal.ChronoUnit.MINUTES).call();

// Reindex to replay the missed mutations into the restored shards.
mgmt = graph.openManagement();
mgmt.updateIndex(mgmt.getGraphIndex("byStatusMixed"), SchemaAction.REINDEX).get();
mgmt.commit();

Where the graph exposes an ingestion timestamp, bound the reconcile to just the outage window at the application layer so the replay is a fraction of a full rebuild:

python 
# Re-emit index mutations only for elements written during the outage window,
# keyed off an application-maintained ingest timestamp property.
from datetime import datetime, timezone

def replay_window(g, start_epoch, end_epoch):
    touched = (
        g.V().has("ingest_ts", __.gte(start_epoch)).has("ingest_ts", __.lte(end_epoch))
        .id().toList()
    )
    # Force a no-op property touch so JanusGraph re-dispatches the index mutation.
    for vid in touched:
        g.V(vid).property("reconciled_at", int(datetime.now(timezone.utc).timestamp())).iterate()
    return len(touched)

Verify: storage ground truth and index document count must converge.

bash 
curl -s -XPOST "http://10.0.2.20:9200/janusgraph_byStatusMixed/_refresh"
STORAGE=$(gremlin -e "g.V().has('account','status','active').count().next()")
INDEX=$(curl -s "http://10.0.2.20:9200/janusgraph_byStatusMixed/_count" | jq .count)
echo "storage=$STORAGE index=$INDEX"   # must agree within tolerance

Fallback and rollback procedures

Validate the search tier at each stage before moving to the graph; these are the recovery paths when a step does not behave.

If Step 1 shows red but all nodes are present. The node did not leave — a shard failed on a live node, often from disk pressure or a corrupt segment. Check _cat/allocation?v for a full disk and free space or raise the watermark before any reroute; reindexing into a full node will only fail again.

If Step 2 reports no_valid_shard_copy on a zero-replica index. There is nothing to reassign; the data is gone from the index. Skip straight to a storage reindex of those shards in Step 4 and raise number_of_replicas to at least 1 so the next node loss self-heals.

If Step 3 will not reach green. Allocation may be blocked by a disk watermark, a version mismatch on the replacement node, or a stuck cluster.routing.allocation.enable=none left over from a rolling restart. Re-enable allocation with a settings update, resolve the watermark, and re-run _cluster/reroute?retry_failed=true. Do not proceed to the reindex against a yellow-with-unassigned-primaries cluster.

If Step 4 parity still fails after the reindex. The outage window was under-estimated and earlier mutations were missed. Widen the window and re-run replay_window, or fall back to a full SchemaAction.REINDEX of the index. Throughout, keep the affected read path on the storage-backed fallback so callers never see the gap — clearing that fallback is the rollback trigger, and it stays set until parity passes. The live-drift mechanics behind an OpenSearch index that keeps diverging are covered in OpenSearch Sync Patterns.