Schema Migration Patterns

A registered JanusGraph type is not editable in place — a property key’s data type and cardinality are frozen the moment mgmt.commit() returns, and no later management transaction can widen a String to a Long or drop an edge label whose data still exists. That immutability is the whole reason live schema changes go wrong: teams reach for a rename or a retype that the Management API silently refuses, or they add an index and route reads to it before it has left REGISTERED. This guide sits under the Graph Schema Validation & Modeling Strategies reference and covers the runtime mechanics of moving a running graph from one shape to another without downtime — the operations you actually run against JanusGraphManagement, in what order, and how to keep every rolling-deploy replica readable throughout. It is the operational complement to Schema Evolution and CI Gating, which governs whether a change is allowed to merge; here the concern is how the approved change lands on production data safely.

The timeline below is the spine of every safe migration on this page: expand the schema additively, migrate the data behind a dual-write, then contract away the old shape once parity holds.

The expand-migrate-contract timeline for an online JanusGraph schema change A left-to-right timeline in three phases. Expand adds the new property key, edge label, or index in a single management transaction while the old shape keeps serving all reads and writes. Migrate turns on dual-write so the application writes both the old and the new shape, then runs an idempotent backfill over existing vertices and verifies parity between the two shapes. Contract cuts reads over to the new shape, stops writing the old one, and disables then removes the superseded index or key. A rollback arrow runs backward from Migrate to Expand, showing that the change is reversible until the contract phase begins. 1 · EXPAND 2 · MIGRATE 3 · CONTRACT add key / label / index one management tx await index ENABLED old shape still serves everything dual-write old + new idempotent backfill verify parity both shapes valid reads still on old cut reads to new stop writing old DISABLE then REMOVE old shape retired point of no return reversible until contract begins — roll back to expand the graph is dual-shaped and fully readable through phases 1 and 2

Core Configuration & Consistency Tuning

Every online migration runs through the distributed schema lock, so the janusgraph.properties values that govern locking and constraint enforcement decide whether a change lands cleanly or wedges the schema mid-flight. The block below is the baseline to migrate against on a CQL backend; the non-default values exist to make a partial mutation recoverable rather than permanent.

properties 
# --- Storage backend ---
storage.backend=cql
storage.hostname=10.0.1.10,10.0.1.11,10.0.1.12
storage.cql.keyspace=janusgraph_prod
storage.cql.local-datacenter=dc1
storage.cql.read-consistency-level=QUORUM
storage.cql.write-consistency-level=QUORUM

# --- Enforce the contract a migration is diffed against ---
schema.default=none
schema.constraints=true

# --- Serialize schema mutations without deadlocking a rolling deploy ---
storage.lock.wait-time=120000
storage.lock.renew-timeout=60000
storage.lock.clean-expired=true

# --- Mixed-index backend (Elasticsearch value also serves OpenSearch) ---
index.search.backend=elasticsearch
index.search.hostname=10.0.2.10,10.0.2.11
index.search.elasticsearch.client-only=true

The constraints that govern a migration, in the order they bite:

  1. schema.default=none makes the new shape opt-in. With auto-creation disabled, a dual-write to a not-yet-registered property key fails loudly instead of silently minting an unindexed key. This is the property that lets the expand phase be the only place new types enter the graph — nothing sneaks in through an application write.
  2. schema.constraints=true freezes the old shape’s type. A registered key’s dataType and Cardinality are immutable; the constraint flag is what turns a retype attempt into a SchemaViolationException at write time rather than corrupt data at read time. Because you cannot edit a key, every retype is really an add-plus-migrate under a new key name.
  3. Lock timeouts must exceed your longest schema transaction and your rolling-deploy window. A 120s wait with a 60s renew lets several replicas roll while one holds the schema lock through an awaitGraphIndexStatus poll. Set them too low and a slow index registration expires the lock mid-mutation, leaving a half-installed index in INSTALLED.
  4. clean-expired=true reclaims a lock a crashed migrator dropped. If the process running the expand phase dies between openManagement() and commit(), the orphaned lock is reclaimed automatically instead of stalling every later deploy on the same keyspace.

The invariant that makes the whole pattern safe is additive-only mutation during expand. Formally, if SoldS_{old} is the set of registered schema elements before the change and SnewS_{new} after, an online-safe expand requires:

Sold⊆SnewS_{old} \subseteq S_{new}

Any change that violates the subset relation — a removal, a rename, a retype — is not an expand and must be decomposed into an additive expand followed by a later contract. Baseline consistency-level selection behind the QUORUM settings is covered in Cassandra Backend Setup, and authoritative parameter definitions live in the official JanusGraph configuration reference.

What JanusGraph Allows, Forbids, and the Workarounds

The Management API divides schema operations into a permissive additive set and a forbidden mutating set, and knowing which is which up front saves you from designing a migration the engine will reject.

Allowed online, no data rewrite:

  • Add a new property key with makePropertyKey(...).dataType(...).cardinality(...).make().
  • Add a new vertex label or edge label with makeVertexLabel(...) / makeEdgeLabel(...).
  • Add a new composite or mixed index over existing keys with buildIndex(...), then REINDEX to populate it.
  • Add a mapped key to an existing mixed index with addIndexKey(index, key).
  • Change a schema element’s name via mgmt.changeName(element, "newName") — this is a metadata rename that preserves the underlying type and data.

Forbidden in place — no management call performs these:

  • Change a registered key’s dataType (for example String → Long).
  • Change a registered key’s Cardinality (for example SINGLE → LIST).
  • Drop a property key, label, or the last key of an index while data referencing it still exists.
  • Retarget an existing index to a different key set.

The workaround for every forbidden operation is the same shape: introduce a new element that has the target property, migrate data onto it, then retire the old one. A retype becomes add userId_v2 as Long, dual-write, backfill, cut over, then disable the index on userId. Note the one real rename escape hatch — mgmt.changeName — only relabels; it never changes type, so it does not help a retype. The immutability rules are exactly the ones a diff enforces in Schema Evolution and CI Gating; this section is what the migration plan those gates demand actually executes.

Here is the expand phase for a retype migration, adding property keys, an edge label, and a composite index in a single management transaction so the change is atomic:

java 
JanusGraphManagement mgmt = graph.openManagement();
try {
    // Retype: userId was String; the new key is the target Long type.
    // You cannot edit the old key, so v2 is a genuinely new element.
    PropertyKey userIdV2 = mgmt.makePropertyKey("userId_v2")
            .dataType(Long.class)
            .cardinality(Cardinality.SINGLE)
            .make();

    // A new edge label the migration also introduces.
    mgmt.makeEdgeLabel("owns_v2").multiplicity(Multiplicity.MULTI).make();

    // A composite index over the new key, built but not yet populated.
    mgmt.buildIndex("byUserIdV2", Vertex.class)
            .addKey(userIdV2)
            .buildCompositeIndex();

    mgmt.commit();   // atomic: all three elements register together or none do
} catch (Exception e) {
    mgmt.rollback(); // a partial expand leaves nothing half-registered
    throw e;
}

// Populate the new composite index over existing data, then block for ENABLED.
mgmt.awaitGraphIndexStatus(graph, "byUserIdV2").status(SchemaStatus.REGISTERED).call();
mgmt = graph.openManagement();
mgmt.updateIndex(mgmt.getGraphIndex("byUserIdV2"), SchemaAction.REINDEX).get();
mgmt.commit();
ManagementSystem.awaitGraphIndexStatus(graph, "byUserIdV2")
        .status(SchemaStatus.ENABLED).call();

Index Synchronization During a Migration

The reason a migration cannot be a single atomic switch is that JanusGraph splits every schema change across two consistency domains, and they converge at different times. Type registration and composite indexes commit synchronously in the storage backend — they are part of the same mutation. A new mixed index on Elasticsearch or OpenSearch is eventually consistent and advances through INSTALLED → REGISTERED → ENABLED on its own schedule. If the contract phase cuts reads to a mixed index still sitting in REGISTERED, the query planner cannot use it and every affected query falls back to a full scan.

The rule that keeps the phases ordered is that reads may only move to an index that has reached ENABLED, and the only reliable way to know is to poll:

java 
// Block server-side until the index is live; bound the wait against row count.
ManagementSystem.awaitGraphIndexStatus(graph, "byUserIdV2")
        .status(SchemaStatus.ENABLED)
        .timeout(10, ChronoUnit.MINUTES)
        .call();

Two lag signals decide whether the migrate phase is progressing or stuck. The first is the wall-clock time an index spends in REGISTERED: a plateau means an index instance is unreachable or a REINDEX job stalled, and the contract phase must not begin. The second is mixed-index queue depth on the org.janusgraph.diskstorage.indexing.IndexProvider bean during the backfill — if the backfill writes faster than the index drains, the eventual-consistency window widens and a parity check run too early reports false mismatches. The reindex mechanics behind populating a new index over live data, and how to recover one that never enables, are the subject of Reindexing & Recovery; the mapping-type decisions that determine whether the new index can even answer the query are in Property Indexing Rules.

Python Integration Pattern

The backfill is the heart of the migrate phase, and it must be idempotent — a distributed job will be retried, replayed, and interrupted, and every one of those must converge on the same result rather than double-writing. The gremlin-python job below streams existing vertices in bounded batches, computes the new-shape value from the old, and writes it only where it is missing, so a re-run over already-migrated data is a no-op. It is safe to stop and resume at any batch boundary.

python 
import logging
from gremlin_python.driver.driver_remote_connection import DriverRemoteConnection
from gremlin_python.process.anonymous_traversal import traversal
from gremlin_python.process.graph_traversal import __
from gremlin_python.process.traversal import P

logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
log = logging.getLogger("backfill")


def backfill_userid_v2(ws_endpoint, batch_size=500):
    conn = DriverRemoteConnection(ws_endpoint, "g")
    g = traversal().withRemote(conn)
    migrated = 0
    try:
        while True:
            # Select a bounded batch of vertices that still lack the new key.
            # has('userId_v2') absence is the idempotency guard: already-migrated
            # vertices drop out of the selection, so a replay does no work.
            batch = (
                g.V().has("userId")
                 .not_(__.has("userId_v2"))
                 .limit(batch_size)
                 .project("vid", "old")
                 .by(__.id_())
                 .by("userId")
                 .toList()
            )
            if not batch:
                break

            for row in batch:
                vid, old_str = row["vid"], row["old"]
                try:
                    new_long = int(old_str)   # the retype: String -> Long
                except (TypeError, ValueError):
                    log.warning("unparseable userId on %s: %r — skipping", vid, old_str)
                    continue
                # Conditional write: coalesce leaves an already-set value untouched,
                # so a retried batch converges instead of clobbering.
                (
                    g.V(vid)
                     .coalesce(
                         __.has("userId_v2"),
                         __.property("userId_v2", new_long),
                     )
                     .iterate()
                )
                migrated += 1

            log.info("migrated=%d (last batch=%d)", migrated, len(batch))
    finally:
        conn.close()
    return migrated

Discipline that keeps the backfill safe under load:

  • The selection predicate is the idempotency key. Filtering on the absence of userId_v2 means the job only ever touches un-migrated vertices, so retries, replays, and parallel workers cannot double-write.
  • Bound every batch. A limit(batch_size) of 200–500 keeps each transaction small enough that a rollback is cheap and the mixed-index dispatch queue does not overflow; an unbounded traversal materializes the whole graph and starves the connection pool. See Connection Pooling for sizing the driver against backfill concurrency.
  • Skip, do not fail, on unconvertible source data. A value that will not parse under the new type is a data-quality finding to log and route, not a reason to abort the whole job — otherwise one bad row blocks the entire migration.
  • Run the backfill only after the expand commit is visible on every replica. A backfill that races the schema registration writes to a key some nodes have not yet seen and throws SchemaViolationException on those nodes.

Deprecating a Property: Dual-Write, Backfill, Cutover

The end-to-end shape of deprecating an old property in favor of a new one strings the phases together into a sequence that keeps every rolling-deploy replica readable throughout:

  1. Expand. Register the new key/index in one management transaction and REINDEX to ENABLED. Old readers and writers are untouched.
  2. Dual-write. Deploy application code that writes both the old and new key on every mutation. Because deploys are rolling, old and new binaries run side by side — the old binary writes only the old key, the new binary writes both, and both are valid because the expand already registered the new key.
  3. Backfill. Run the idempotent job above to populate the new key on vertices written before dual-write began. After it completes, every vertex carries both shapes.
  4. Verify parity. Assert that new-shape and old-shape values agree across a sample (or the full set) before trusting the new key. A mismatch here means either the backfill transform is wrong or a dual-writer regressed.
  5. Cut over reads. Deploy code that reads the new key. Reads and writes now both use the new shape; the old key is still written but no longer read.
  6. Contract. Stop writing the old key, then DISABLE_INDEX and finally REMOVE_INDEX on the old index, and drop the old key once nothing references it.

The forwards/backwards compatibility that makes step 2 survivable is the crux: during a rolling deploy the fleet is mixed, so the new shape must be additive (old readers ignore an unknown key) and the old shape must remain fully written (new readers still find it) until the entire fleet has advanced. Never cut reads to the new shape while any replica still writes only the old one — that replica produces vertices the new readers cannot see. The step-by-step runbook for phases 1 through 5 is Performing Online Schema Migrations; when a phase goes wrong, the recovery sequence is Rolling Back a Failed Schema Change.

The contract phase uses the index teardown primitives. Disable first so the planner stops routing to the index, then remove to reclaim its storage:

java 
JanusGraphManagement mgmt = graph.openManagement();
JanusGraphIndex legacy = mgmt.getGraphIndex("byUserId");
// Step A: DISABLE_INDEX removes the index from query planning immediately.
mgmt.updateIndex(legacy, SchemaAction.DISABLE_INDEX).get();
mgmt.commit();
ManagementSystem.awaitGraphIndexStatus(graph, "byUserId")
        .status(SchemaStatus.DISABLED).call();

// Step B: REMOVE_INDEX reclaims the index data after it is disabled.
mgmt = graph.openManagement();
mgmt.updateIndex(mgmt.getGraphIndex("byUserId"), SchemaAction.REMOVE_INDEX).get();
mgmt.commit();

// updateSchemaElement can relabel a leftover element as deprecated so audits
// flag it, without touching its immutable type.
mgmt = graph.openManagement();
PropertyKey oldKey = mgmt.getPropertyKey("userId");
mgmt.changeName(oldKey, "userId_deprecated");
mgmt.commit();

Diagnostics & Operational Fallbacks

Expose the schema and lock beans through the JMX-to-Prometheus exporter, then triage against these symptom/diagnosis/resolution triplets — the three that account for most stalled migrations.

  • Symptom: The expand transaction hangs, and later rolling-deploy replicas time out opening management transactions. Diagnosis: schema-lock contention — a prior migrator died holding the distributed lock on the schema keyspace, visible as lock rows on the storage log and a rising storage.lock wait on the CQLStoreManager bean. Resolution: wait out storage.lock.wait-time, or with clean-expired=true let the orphaned lock reclaim; confirm no zombie migration process is still connected before retrying the expand, and never lower the lock timeout to “fix” it — that trades a stall for a half-applied mutation.

  • Symptom: mgmt.printIndexes() shows the new index pinned at REGISTERED, and the backfill’s writes never become queryable. Diagnosis: stuck index status — an index instance is unreachable or a REINDEX job stalled, so the index cannot advance to ENABLED. Resolution: confirm every JanusGraph instance is up; if one is dead, drop its stale registration with mgmt.forceCloseInstance(...), then re-run SchemaAction.REINDEX and re-poll. Do not begin the contract phase — cutting reads to a REGISTERED index forces full scans.

  • Symptom: A second openManagement() in the same migration throws or blocks, and schema reads return stale definitions. Diagnosis: an open management transaction was never committed or rolled back — a leaked JanusGraphManagement handle holds the lock and caches a stale view. Resolution: always wrap the mutation in try/commit/catch-rollback and never open a second management transaction before the first has closed; if a handle leaked, mgmt.rollback() it and force cluster-wide cache invalidation before reopening.

  • Symptom: Writes begin failing with SchemaViolationException immediately after the dual-write deploy. Diagnosis: the application wrote the new key before the expand commit was visible on that replica, or the new key’s type does not match the value being written. Resolution: confirm the expand transaction committed and propagated to every node, verify the dual-writer’s type matches the registered dataType, then replay the failed batch — it is safe because the backfill selection guards on absence.

  • Symptom: Parity check reports mismatches that shrink on re-run. Diagnosis: the mixed-index eventual-consistency window — the parity read ran before the index drained the backfill writes. Resolution: read the parity sample through a storage-backed id lookup rather than a has() predicate, or wait for the IndexProvider queue to drain before sampling; a mismatch that disappears on retry was never a real divergence.

Route these deterministically when they fire — a stalled REGISTERED index during a migration is a page, not a warning — through the policy in Alert Routing for Violations, and validate the mutation against the invariants in Vertex and Edge Validation before you begin.

Frequently Asked Questions

Can I change a JanusGraph property key’s data type in place? No. A registered key’s dataType and Cardinality are immutable once committed, and no Management API call edits them. The supported path is to add a new key with the target type, dual-write both keys, backfill existing vertices with an idempotent job, cut reads to the new key, and then disable and remove the old index. mgmt.changeName renames an element but never changes its type, so it does not help a retype.

Why does the migration need a dual-write phase instead of a single cutover? Because production deploys are rolling, so old and new application binaries run side by side for a window. Dual-write keeps both shapes valid during that window: old binaries write the old key, new binaries write both, and reads stay on the old shape until every replica has advanced and the backfill has populated the new key everywhere. A single cutover would leave vertices written by old binaries invisible to new readers.

How do I add an index to a live graph without downtime? Build the index in a management transaction, wait for it to reach REGISTERED, run SchemaAction.REINDEX to populate it over existing data, then poll awaitGraphIndexStatus until it is ENABLED before routing any read to it. The old query path keeps working the entire time because the new index is additive; only when it is ENABLED does the planner start using it.

What is the difference between DISABLE_INDEX and REMOVE_INDEX? DISABLE_INDEX takes the index out of query planning immediately but leaves its data in place, which makes it instantly reversible — you re-enable it if the cutover regresses. REMOVE_INDEX reclaims the index’s backend storage and is terminal. Always disable first, confirm the new path is healthy, and only then remove, so the contract phase has a fast rollback until the final step.

My backfill job died halfway — is it safe to just run it again? Yes, if it is written idempotently. The job selects only vertices that lack the new key and writes conditionally via coalesce, so already-migrated vertices drop out of the selection and a re-run does no work on them. That absence-based guard is what makes the backfill safe to stop, resume, and parallelize without double-writing.