Running a Zero-Downtime Reindex

Rebuilding a JanusGraph mixed index usually reads like a maintenance-window job, but with the read path routed through storage-backed id lookups it becomes a live operation that never returns a wrong answer to a caller. This procedure sits under Reindexing and Recovery and narrows it to a single task: populating a registered index while production queries keep flowing, then cutting reads over to the index only after it is provably complete. The specific failure it prevents is the mid-rebuild wrong answer — the interval where an index is REGISTERED but not yet ENABLED, half-populated, and a naive read path trusts it and drops durable vertices from the result. The whole design is a discipline of not trusting the index until the job proves it is safe.

Read path routing during a zero-downtime reindex before and after the ENABLED cutover During the reindex, the application read path is routed to storage-backed id lookups against Cassandra or ScyllaDB, which always return complete results, while the reindex job populates the mixed index in the Elasticsearch or OpenSearch backend in the background. A parity gate compares the storage count against the index document count; only when the index reaches ENABLED and parity passes does the read path flip to serving queries from the index. Before the cutover the half-built index is never read. Read path application query parity + ENABLED gate flip only when both pass Storage id lookup Cassandra / ScyllaDB · durable Mixed index ES / OS · populating in background during reindex after ENABLED + parity REINDEX job ↓ populates
Reads serve from storage-backed id lookups while the index populates in the background; the path flips to the index only after it reaches ENABLED and a parity check passes.

Prerequisites

Confirm every item before touching the management API. The most common cause of a “zero-downtime” reindex that still drops results is a read path that was never actually rerouted to storage before the job started.

  • JanusGraph 0.6.x or 1.0.x against a CQL backend (Cassandra 3.11+/4.x or ScyllaDB) with an Elasticsearch or OpenSearch index backend already reachable and index.search.backend configured.
  • The index already declared in the schema and sitting at INSTALLED or REGISTERED — this procedure populates an existing index definition, it does not create one.
  • A storage-backed read fallback in the application: reads for the affected predicate must be answerable by id lookup or full scan against storage, not only through the index. If storage is not yet the source of truth, stand it up via Cassandra Backend Setup first.
  • Management-API access from a JanusGraph instance and a maintenance path to run either an in-process reindex or a MapReduce job.
  • A metrics scrape (metrics.enabled=true) plus direct access to the index backend _count and _cat/thread_pool endpoints, so drift and parity are measured rather than assumed.
  • Known cluster state. All JanusGraph instances registered (mgmt.getOpenInstances() shows no stale entries) and all storage nodes UN in nodetool status.

Step 1 — Route reads to storage-backed lookups

Before the index changes state at all, move the affected read path off the index and onto storage. This is what makes the rest of the procedure safe: while the index is incomplete, no caller ever consults it.

python 
# Feature-flag the read path. While reindexing is in progress, resolve the
# predicate from storage (id lookup / full scan), never from the mixed index.
REINDEX_IN_PROGRESS = True

def find_active(g, status="active"):
    if REINDEX_IN_PROGRESS:
        # Storage-backed: correct but expensive. Bypasses the mixed index.
        return g.V().has("account", "status", status).valueMap().toList()
    # Index-backed hot path, re-enabled only after cutover in Step 5.
    return g.V().has("status", status).valueMap().toList()

Verify: confirm the flag is live and the index is not on the read path.

bash 
# No mixed-index query profile should appear for the affected predicate.
grep -c "backend-query" /var/log/janusgraph/janusgraph.log
# Expect the storage-scan path in a profile() of the live query, not an index hit.

Step 2 — Register the index and gate on REGISTERED

Advance the index to REGISTERED on every instance. The awaitGraphIndexStatus gate is mandatory — if one instance never acknowledges, a later reindex reports success while that instance keeps writing to an index it does not know exists.

java 
JanusGraphManagement mgmt = graph.openManagement();
mgmt.updateIndex(mgmt.getGraphIndex("byStatusMixed"), SchemaAction.REGISTER_INDEX);
mgmt.commit();

ManagementSystem.awaitGraphIndexStatus(graph, "byStatusMixed")
    .status(SchemaStatus.REGISTERED)
    .timeout(10, java.time.temporal.ChronoUnit.MINUTES)
    .call();

Verify: the index must report REGISTERED, not INSTALLED.

groovy 
// Gremlin console
mgmt = graph.openManagement()
println mgmt.getGraphIndex('byStatusMixed').getIndexStatus(
    mgmt.getPropertyKey('status'))
mgmt.rollback()

Step 3 — Run the reindex, throttled

Populate the registered index. Choose in-process for graphs that fit one JVM and MapReduce for anything larger. Either way, keep index refresh off so the job batches writes instead of forcing a Lucene refresh that competes with live query shards.

properties 
# Apply before launching the job so the reindex does not starve live traffic.
index.search.elasticsearch.bulk-refresh=false
java 
// In-process reindex — for graphs that fit a single JVM's scan budget.
JanusGraphManagement mgmt = graph.openManagement();
mgmt.updateIndex(mgmt.getGraphIndex("byStatusMixed"), SchemaAction.REINDEX).get();
mgmt.commit();

For a large graph, fan the scan across the storage cluster and cap the mapper concurrency so the job never consumes the whole storage tier:

bash 
# MapReduce reindex with a bounded mapper limit as the throttle.
export HADOOP_OPTS="-Dmapreduce.job.running.map.limit=8"
janusgraph-reindex.sh --index byStatusMixed --index-type mixed

Verify: the index document count must be climbing toward the storage count.

bash 
# Watch the index fill. The count should rise, then plateau near storage truth.
watch -n 10 'curl -s "http://10.0.2.20:9200/janusgraph_byStatusMixed/_count" | jq .count'

Step 4 — Poll awaitGraphIndexStatus for ENABLED

Do not flip reads on a timer. Block on the actual status transition; the reindex may take hours on a large graph, and ENABLED is the only signal that the job has finished populating.

java 
ManagementSystem.awaitGraphIndexStatus(graph, "byStatusMixed")
    .status(SchemaStatus.ENABLED)
    .timeout(6, java.time.temporal.ChronoUnit.HOURS)
    .call();

If the index was already ENABLED before the reindex (a rebuild of an existing index rather than a first build), watch the index count converge on the storage count instead of the status transition, because status will not change.

Verify: confirm the terminal status.

groovy 
mgmt = graph.openManagement()
println mgmt.getGraphIndex('byStatusMixed').getIndexStatus(
    mgmt.getPropertyKey('status'))   // must print ENABLED
mgmt.rollback()

Step 5 — Verify parity, then flip reads

Prove the index is complete before trusting it. Refresh the index, compare the storage ground-truth count against the index document count, and only then clear the feature flag so reads return to the index.

bash 
# Force the last batched writes visible, then compare the two authorities.
curl -s -XPOST "http://10.0.2.20:9200/janusgraph_byStatusMixed/_refresh"
STORAGE=$(gremlin -e "g.V().has('account','status','active').count().next()")
INDEX=$(curl -s "http://10.0.2.20:9200/janusgraph_byStatusMixed/_count" | jq .count)
echo "storage=$STORAGE index=$INDEX"

When STORAGE and INDEX agree within tolerance, clear REINDEX_IN_PROGRESS so the hot path serves from the index again. The routing that decides which backing index the flipped read consults is covered in Mixed-Index Routing.

Verify: the index-backed query must now match the storage-backed count.

bash 
grep -c "backend-query" /var/log/janusgraph/janusgraph.log   # index path active again

Fallback and rollback procedures

Validate between steps rather than stacking changes; every step has a defined recovery path.

If Step 1 leaves reads on the index. Any caller still hitting the mixed index during the rebuild can get a partial result. Stop the reindex, confirm the flag is enforced across every application instance, and only resume once a profile() of the live query shows the storage-scan path.

If Step 2 will not reach REGISTERED. A stale instance is blocking the transition. Run mgmt.getOpenInstances(), force-close the offender with mgmt.forceCloseInstance(id), then re-issue REGISTER_INDEX. Never proceed to a reindex while any instance is unregistered.

If Step 3 fails with OutOfMemoryError. The in-process reader exceeded heap. Abort, switch to the MapReduce path, and if MapReduce is unavailable shard the reindex by key range into bounded passes. The rebuild is idempotent, so a failed pass can be re-run.

If Step 4 times out. The reindex is still running or wedged. Re-poll with a longer timeout if the index count is still climbing; if the count is flat and well below storage truth, treat it as a stalled job — inspect the index backend _cat/thread_pool/write?v queue for rejections and re-run the reindex after clearing the cause.

If Step 5 parity fails after cutover. Immediately set REINDEX_IN_PROGRESS = True again to send reads back to storage — this is the whole-procedure rollback and it is instant. Then re-run a targeted reindex over the divergent window and re-verify before flipping again. Because the read path never depended on the index being correct, this rollback loses no data and returns no wrong answers.