Reindexing and Recovery

A JanusGraph mixed or composite index only answers queries for the data that existed when it was built; every mutation committed before the index reached ENABLED, and every write the index backend silently dropped, stays invisible until you rebuild. This reference sits under External Index Synchronization & Consistency Tuning and isolates one operational discipline: driving the reindex lifecycle and reconciling an index that has diverged from storage. The failure mode it prevents is a graph that returns fewer results than it holds — a query planner that trusts an index which is stale, half-built, or missing shards, so has() predicates and order().by() steps quietly omit durable vertices. Everything below treats the index as a derived, rebuildable projection of the storage rows, and treats ENABLED as the only status a read path is allowed to trust.

The reindex lifecycle is a state machine. A newly declared index is INSTALLED; it must be registered on every JanusGraph instance to reach REGISTERED, populated by a reindex job, and only then transitioned to ENABLED. Drift — a divergence between the storage row count and the index document count — is the trigger that forces you back through it.

Core Configuration & Consistency Tuning

Reindexing is a schema-management operation, not a data mutation, so it runs through the JanusGraphManagement API and the standalone MapReduceIndexManagement / IndexRepairJob machinery — never through a traversal. The configuration that governs it lives in two places: the index-backend section of janusgraph.properties, which decides how aggressively the reindex writes, and the transaction settings that decide whether a partially built index can leak into reads.

# Index backend the reindex writes into
index.search.backend=elasticsearch
index.search.hostname=10.0.2.20,10.0.2.21,10.0.2.22
index.search.elasticsearch.client-only=true

# Reindex write shaping — bound the bulk pressure the job applies
index.search.elasticsearch.bulk-refresh=false
index.search.elasticsearch.retry_on_conflict=3
index.search.max-result-set-size=100000

# Storage-side scan batching for the reindex reader
storage.cql.read-consistency-level=LOCAL_QUORUM
storage.batch-loading=false
ids.block-size=100000

Operational constraints, in the order they bite during a rebuild:

1. bulk-refresh=false keeps the reindex from starving live traffic. With per-request refresh disabled, the index backend batches segment writes instead of forcing a Lucene refresh on every bulk call. Leaving it at the default forces a refresh storm that competes directly with query traffic for the same shards. Re-enable refresh (or issue an explicit _refresh) only after the job completes and before you compare counts.
2. storage.batch-loading must stay false during a reindex of a live graph. Batch loading relaxes consistency checks and skips locking; that is correct for an initial cold load but corrupts a rebuild running alongside live mutations, because concurrent writes can be silently overwritten.
3. ids.block-size throttles ID-block contention. The reindex reader allocates ID blocks as it scans; an undersized block forces frequent round-trips to the ID allocator and slows the scan while inflating storage load. Size it to the reindex batch, not the steady-state write rate.
4. max-result-set-size caps the per-shard fetch during verification. It does not affect the write path, but a too-small value makes the post-reindex parity query silently truncate and report false drift.

The reindex reader scans the full storage keyspace, so its wall-clock cost is a function of vertex count and scan parallelism, not index size:

$$t_{\text{reindex}} \approx \frac{N_{\text{elements}}}{P_{\text{scan}} \times r_{\text{batch}}}$$

where $N_{\text{elements}}$ is the vertex-plus-edge count in scope, $P_{\text{scan}}$ the number of parallel scan partitions (mapper count under MapReduce, thread count in-process), and $r_{\text{batch}}$ the per-partition batch throughput. The equation is why an in-process reindex on a billion-element graph is impractical: a single JVM caps $P_{\text{scan}}$ at its core count, whereas a MapReduce job spreads the scan across the whole storage cluster. The mixed-index routing that decides which backing index serves a query — and therefore which index a stale rebuild will poison — is covered in Mixed-Index Routing.

Index Synchronization Window & Drift Detection

The synchronization window is the gap between a durable storage commit and the moment that commit is queryable through the index. A reindex does not close that window for new writes — it closes the historical gap for writes that predate the index or were lost. The two are different problems, and conflating them is how a “successful” reindex still returns stale results: the job rebuilt history but never accounted for the mutations that landed while it ran.

Detect drift by comparing the two authorities directly rather than trusting the index to report its own health:

• Storage count — the ground truth. Count elements matching the indexed predicate straight from a traversal that bypasses the mixed index, e.g. a full-scan g.V().has('status','active').count() executed with the index profile forced off, or the storage-backend row count for the relevant column family.
• Index document count — what the index believes it holds. Query the backend directly: GET /janusgraph_status/_count against Elasticsearch or OpenSearch for the mapped index name.
• IndexProvider queue depth — the leading indicator. A rising _cat/thread_pool/write?v queue on the index backend, or climbing JanusGraph index backpressure counters, means mutations are being enqueued faster than they apply. Sustained backpressure guarantees drift even before the counts diverge.

A non-zero, stable difference is expected — it is the in-flight sync window. A difference that grows is drift and demands a targeted reindex over the affected time window. The acknowledgment-boundary decision behind how long a read should wait for the index before it is considered consistent is detailed in Eventual vs Strong Consistency; the backend-specific mechanics of catching a drifting OpenSearch index up live are in OpenSearch Sync Patterns.

Management API Reindex Lifecycle

The lifecycle is driven entirely from the management API. The annotated block below registers an existing INSTALLED index, waits for every instance to acknowledge, runs the reindex, and gates the transition to ENABLED on the reindex actually finishing. Read the comments — the ordering is the whole point, and skipping the awaitGraphIndexStatus gate is the single most common cause of a reindex that reports success while half the JanusGraph instances never registered.

import org.janusgraph.core.schema.JanusGraphManagement;
import org.janusgraph.core.schema.SchemaAction;
import org.janusgraph.core.schema.SchemaStatus;
import org.janusgraph.graphdb.database.management.ManagementSystem;
import org.janusgraph.core.schema.JanusGraphIndex;
import java.time.Duration;

// 1. REGISTER: move the index from INSTALLED to REGISTERED on every instance.
//    This must succeed cluster-wide before any reindex is meaningful.
JanusGraphManagement mgmt = graph.openManagement();
JanusGraphIndex idx = mgmt.getGraphIndex("byStatusMixed");
mgmt.updateIndex(idx, SchemaAction.REGISTER_INDEX);
mgmt.commit();

// 2. GATE: block until the index is REGISTERED on ALL instances, or bail.
//    awaitGraphIndexStatus polls until the target status or the timeout.
ManagementSystem.awaitGraphIndexStatus(graph, "byStatusMixed")
    .status(SchemaStatus.REGISTERED)
    .timeout(10, java.time.temporal.ChronoUnit.MINUTES)
    .call();

// 3. REINDEX: populate the registered index from the storage rows.
//    In-process for small graphs; use MapReduceIndexManagement for large ones.
mgmt = graph.openManagement();
mgmt.updateIndex(mgmt.getGraphIndex("byStatusMixed"), SchemaAction.REINDEX).get();
mgmt.commit();

// 4. GATE on ENABLED: only now may the query planner use the index.
//    Reads gated on this status will never see a half-built index.
ManagementSystem.awaitGraphIndexStatus(graph, "byStatusMixed")
    .status(SchemaStatus.ENABLED)
    .timeout(6, java.time.temporal.ChronoUnit.HOURS)
    .call();

For a graph too large to scan in one JVM, replace step 3 with the MapReduce path, which fans the storage scan across the storage cluster instead of one process:

import org.janusgraph.hadoop.MapReduceIndexManagement;

MapReduceIndexManagement mr = new MapReduceIndexManagement(graph);
// Runs a Hadoop job whose mapper count sets the scan parallelism P_scan.
mr.updateIndex(mgmt.getGraphIndex("byStatusMixed"), SchemaAction.REINDEX).get();

Choose in-process only when the element count fits comfortably in a single JVM’s scan budget (roughly tens of millions); above that, MapReduce is not an optimization but a requirement, because the in-process reader will OOM on the intermediate result set long before it finishes.

Python Integration Pattern

Operators verify a reindex from Python, not Java — the management API triggers the job, but the parity check that decides whether it is trustworthy belongs in the same tooling that runs the read path. The gremlin-python client below gates reads on ENABLED and refuses to flip traffic until the storage-derived count and the index-derived count agree within a tolerance. The discipline is that a read must never be served from an index whose status the client has not confirmed.

from gremlin_python.driver.driver_remote_connection import DriverRemoteConnection
from gremlin_python.process.anonymous_traversal import traversal
from gremlin_python.process.graph_traversal import __
import logging

logger = logging.getLogger(__name__)


class ReindexVerifier:
    def __init__(self, ws_endpoint: str, index_name: str):
        self.conn = DriverRemoteConnection(ws_endpoint, "g")
        self.g = traversal().withRemote(self.conn)
        self.index_name = index_name

    def storage_truth_count(self, prop: str, value) -> int:
        # Full-scan count that bypasses the mixed index: this is ground truth.
        # It is deliberately expensive — run it only during verification, never
        # on the hot path.
        return (
            self.g.V().has(prop, value)
            .count().next()
        )

    def parity_ok(self, prop, value, index_doc_count: int, tolerance: int = 0) -> bool:
        # index_doc_count is fetched out-of-band from the ES/OS _count API.
        # A stable non-zero gap is the sync window; a growing gap is drift.
        truth = self.storage_truth_count(prop, value)
        delta = abs(truth - index_doc_count)
        if delta > tolerance:
            logger.warning(
                "drift on %s: storage=%d index=%d delta=%d",
                self.index_name, truth, index_doc_count, delta,
            )
            return False
        return True

    def read_via_index(self, prop, value):
        # Callers invoke this ONLY after parity_ok returns True and the index
        # status is confirmed ENABLED. Otherwise fall back to storage_truth_count.
        return self.g.V().has(prop, value).valueMap().toList()

    def close(self):
        self.conn.close()

Implementation requirements:

• storage_truth_count must force a full scan, not a has() that the planner satisfies from the very index under test — otherwise you are comparing the index against itself and drift is undetectable.
• index_doc_count comes from the backend _count endpoint, kept out of the traversal so a broken index cannot fabricate a passing number.
• read_via_index is guarded: callers gate on both ENABLED status and parity_ok, and fall back to the storage-backed count when either fails. This is the same storage-backed-lookup fallback that keeps reads correct during the rebuild described in Running a Zero-Downtime Reindex.

Throttling Reindex Against Live Traffic

A reindex on a production graph shares storage read bandwidth, index write bandwidth, and JVM heap with live queries. Left unthrottled it starves them. The controls that matter are the scan parallelism, the index bulk shaping, and — for MapReduce — the mapper concurrency.

• Cap scan parallelism to leave headroom. For an in-process reindex, the scan thread count is bounded by storage.cql read capacity; keep it below the point where storage read latency for live traffic degrades. For MapReduce, set mapreduce.job.running.map.limit so the scan never consumes the whole storage cluster at once.
• Keep index refresh off during the job. index.search.elasticsearch.bulk-refresh=false prevents the reindex from forcing a Lucene refresh that competes with query shards; the trade is that newly written documents are not searchable until the next refresh interval, which is exactly what you want during a rebuild.
• Bound the bulk queue. Size index.search.elasticsearch.bulk-size and the backend thread_pool.write.queue_size so reindex bulk requests queue rather than reject, but not so deep that live writes wait behind a wall of reindex batches.
• Run the heaviest rebuilds off-peak. The MapReduce path lets you schedule a full reindex during a low-traffic window and checkpoint it, rather than holding a single JVM open for hours.

Reindex throttling and connection-pool sizing interact: the reindex reader draws from the same CQL pool as live traffic, so a rebuild launched against an undersized pool manifests as thread starvation on the query path, not on the job. The pool sizing that keeps the two from colliding is covered in Connection Pooling.

Diagnostics & Operational Fallbacks

Expose index status through mgmt.printIndexes() and the backend _cat/indices?v / _cluster/health endpoints, then triage against these symptom/diagnosis/resolution triplets.

• Symptom: an index sits at INSTALLED or REGISTERED forever; the reindex never starts. Diagnosis: one JanusGraph instance never acknowledged the registration — a stale instance holds an open management transaction or was never restarted after the schema change, so awaitGraphIndexStatus for REGISTERED never returns. Resolution: list open instances with mgmt.getOpenInstances(), force-close the stale one with mgmt.forceCloseInstance(id), then re-issue REGISTER_INDEX and re-poll. Never ENABLE an index that has not reached REGISTERED cluster-wide.

• Symptom: queries return partial results — some matching vertices appear, others do not — even though storage holds them all. Diagnosis: the index was enabled before the reindex finished, or a concurrent write landed during the rebuild with bulk-refresh off and was never refreshed. The index is authoritative for the planner but incomplete. Resolution: re-run a targeted REINDEX over the affected key range, issue an explicit _refresh, and re-verify parity before trusting reads. Until parity passes, route the affected predicate through the storage-backed full-scan fallback.

• Symptom: the in-process reindex fails with an OutOfMemoryError partway through the scan. Diagnosis: the single-JVM reader accumulated an intermediate result set larger than heap — the graph is past the in-process ceiling, or max-result-set-size is unbounded. Resolution: switch to MapReduceIndexManagement, which streams the scan across mappers instead of buffering in one heap. If MapReduce is unavailable, shard the reindex by key range and run bounded passes, lowering ids.block-size and the batch size per pass.

• Symptom: live query P99 spikes the moment a reindex starts. Diagnosis: the reindex is starving live traffic — either refresh is on and competing for shards, or the scan is consuming the whole CQL read pool. Resolution: set bulk-refresh=false, cap scan parallelism and mapper concurrency, and confirm the reindex reader is not exhausting the connection pool the query path shares.

• Symptom: storage count and index document count diverge by a growing margin after an index-backend restart. Diagnosis: the index backend lost documents during a node or shard failure and the mutations that landed during the outage were never applied — classic post-failure drift. Resolution: restore replicas or the failed node, then run a targeted reindex bounded to the outage window rather than a full rebuild, following Recovering from an Index Node Failure.

Related

• Up a level: External Index Synchronization & Consistency Tuning — the storage-to-index boundary this rebuild discipline restores.
• Running a Zero-Downtime Reindex — the step-by-step procedure for rebuilding an index while serving reads from storage-backed lookups.
• Recovering from an Index Node Failure — assessing red/yellow cluster health and reconciling the index after an Elasticsearch or OpenSearch node loss.
• Mixed-Index Routing — how the planner decides which backing index answers a query, and which one a stale rebuild poisons.
• OpenSearch Sync Patterns — backend-specific mechanics for catching a drifting index up live.
• Eventual vs Strong Consistency — the acknowledgment boundary that defines how long a read waits for the index before it is consistent.