Performing Online Schema Migrations

This procedure walks a single schema change through the graph while it stays fully readable and writable — no maintenance window, no read outage, no lost writes on the replicas still running the old binary. It sits under the Schema Migration Patterns reference and narrows that pattern to one concrete task: migrating a property from an old shape to a new one using expand-migrate-contract, with a command and a verification at every step. The specific failure it prevents is the mid-migration read hole — the state where a cutover races ahead of the backfill and vertices written before the change go silently missing from queries that now expect the new key. Every step below is reversible until the final contract, and each ends with a Verify: you must pass before continuing.

The five-step online migration sequence with the dual-shape window highlighted Five sequential steps left to right. Step one expands the schema by registering the new key and index in one management transaction. Step two turns on dual-write so the application writes both the old and new key. Step three runs an idempotent backfill over vertices written before dual-write. Step four verifies parity between the two shapes. Step five contracts by cutting reads to the new key and disabling the old index. A shaded band under steps two through four marks the dual-shape window, during which both shapes are valid and reads remain on the old key, making the whole span reversible. dual-shape window · both keys valid · reads still on old · reversible Step 1 expand schema Step 2 dual-write Step 3 backfill Step 4 verify parity Step 5 cut over + contract the graph never stops serving reads or writes
Steps 2 through 4 run inside the dual-shape window; only step 5 is a point of no return.

Prerequisites

Confirm every item before touching production. Skipping the fleet-visibility check is the most common cause of a migration that throws SchemaViolationException on a subset of nodes.

  • JanusGraph 0.6.x or 1.0.x on a CQL backend (Cassandra 3.11+/4.x or ScyllaDB), with schema.default=none and schema.constraints=true already set so the new key must be explicitly registered before any write reaches it.
  • gremlinpython matching your server’s TinkerPop line (3.5.x for JG 0.6, 3.6.x for JG 1.0) for the backfill and verification jobs.
  • A rollout mechanism that supports two application deploys — one to enable dual-write, one to cut over reads — because the migration is deliberately split across them.
  • Write access to run Management API scripts through the Gremlin console or a service account, and a JMX or Prometheus scrape so index status and lag are measurable, not guessed.
  • Known cluster state. nodetool status must show UN for all storage nodes, and the mixed-index backend must be healthy — a migration cannot populate a new index against a degraded Elasticsearch or OpenSearch cluster. If the change adds a mixed index, review Property Indexing Rules for the mapping type first, since a wrong mapping cannot be edited later.

Step 1 — Expand the schema in one management transaction

Register every new element the migration needs — the new property key, any new label, and its index — atomically, so a partial expand cannot leave the schema half-defined. This example retypes a String userId to a Long userId_v2 and indexes it.

python 
from gremlin_python.driver.client import Client

EXPAND = """
mgmt = graph.openManagement()
try {
    if (!mgmt.containsPropertyKey('userId_v2')) {
        k = mgmt.makePropertyKey('userId_v2').dataType(Long.class)
               .cardinality(org.janusgraph.core.Cardinality.SINGLE).make()
        mgmt.buildIndex('byUserIdV2', Vertex.class).addKey(k).buildCompositeIndex()
    }
    mgmt.commit(); 'COMMITTED'
} catch (Exception e) { mgmt.rollback(); throw e }
"""

client = Client("ws://janusgraph-server:8182/gremlin", "g")
print(client.submit(EXPAND).all().result())
client.close()

The containsPropertyKey guard makes the expand idempotent, so a retried deploy is a no-op rather than a duplicate-registration error. Once the definition is registered, populate the index over existing rows and block until it is live:

python 
REINDEX = """
import org.janusgraph.core.schema.SchemaAction
import org.janusgraph.core.schema.SchemaStatus
import org.janusgraph.graphdb.database.management.ManagementSystem
mgmt = graph.openManagement()
mgmt.updateIndex(mgmt.getGraphIndex('byUserIdV2'), SchemaAction.REINDEX).get()
mgmt.commit()
ManagementSystem.awaitGraphIndexStatus(graph, 'byUserIdV2').status(SchemaStatus.ENABLED).call()
              .getSucceeded()
"""
client = Client("ws://janusgraph-server:8182/gremlin", "g")
print(client.submit(REINDEX).all().result())
client.close()

Verify: the new index reports ENABLED and the old query path is untouched.

groovy 
mgmt = graph.openManagement()
println mgmt.getGraphIndex('byUserIdV2').getIndexStatus(mgmt.getPropertyKey('userId_v2'))
// expect: ENABLED

Step 2 — Turn on dual-write from the application

Deploy application code that writes both the old and the new key on every mutation of the affected vertices. Because the deploy is rolling, old binaries keep writing only the old key while new binaries write both — and both are valid, because Step 1 already registered the new key across the fleet.

python 
def write_account(g, user_id_str: str, region: str):
    user_id_long = int(user_id_str)
    # Dual-write: the old String key and the new Long key are both set.
    # Old readers ignore userId_v2; new readers are not live yet.
    (
        g.V().has("userId", user_id_str).fold()
         .coalesce(__.unfold(), __.addV("account").property("userId", user_id_str))
         .property("userId_v2", user_id_long)
         .property("region", region)
         .iterate()
    )

Deploy this to the whole fleet before proceeding. Do not advance until every replica writes both keys, or Step 3’s parity assumption breaks.

Verify: vertices written after the deploy carry both keys.

python 
sample = g.V().has("userId_v2").limit(20).valueMap("userId", "userId_v2").toList()
assert all("userId" in v and "userId_v2" in v for v in sample), "dual-write incomplete"
print(f"dual-write confirmed on {len(sample)} recent vertices")

Step 3 — Backfill existing data with an idempotent job

Populate the new key on vertices written before dual-write began. The job selects only vertices missing the new key and writes conditionally, so it is safe to stop, resume, and parallelize — a re-run does no work on already-migrated vertices.

python 
from gremlin_python.process.graph_traversal import __

def backfill(g, batch_size=500):
    migrated = 0
    while True:
        batch = (
            g.V().has("userId").not_(__.has("userId_v2"))
             .limit(batch_size)
             .project("vid", "old").by(__.id_()).by("userId")
             .toList()
        )
        if not batch:
            break
        for row in batch:
            try:
                new_val = int(row["old"])
            except (TypeError, ValueError):
                print(f"skip unparseable userId on {row['vid']}: {row['old']!r}")
                continue
            (
                g.V(row["vid"])
                 .coalesce(__.has("userId_v2"), __.property("userId_v2", new_val))
                 .iterate()
            )
            migrated += 1
        print(f"migrated so far: {migrated}")
    return migrated

Verify: no un-migrated vertices remain (ignoring intentionally skipped rows).

python 
remaining = g.V().has("userId").not_(__.has("userId_v2")).count().next()
print(f"vertices still missing userId_v2: {remaining}")
# expect 0, or exactly the count of logged unparseable rows

Step 4 — Verify parity between the two shapes

Before trusting the new key, prove that the new-shape value agrees with the old-shape value everywhere. Read the sample through the storage-backed id path rather than a fresh has('userId_v2') predicate so a lagging mixed index does not produce false mismatches.

python 
def verify_parity(g, sample_size=5000):
    rows = (
        g.V().has("userId_v2").limit(sample_size)
         .project("old", "new").by("userId").by("userId_v2")
         .toList()
    )
    mismatches = [r for r in rows if str(r["new"]) != str(r["old"])]
    print(f"checked={len(rows)} mismatches={len(mismatches)}")
    return mismatches

bad = verify_parity(g)
assert not bad, f"parity failed on {len(bad)} vertices — do not cut over"

Verify: the assertion passes with zero mismatches. If mismatches shrink on re-run they were index lag, not divergence; wait for the IndexProvider queue to drain and re-check. A stable non-zero count is a real transform bug — stop and follow Rolling Back a Failed Schema Change.

Step 5 — Cut over reads, then contract

Only now move reads to the new key. Deploy application code that reads userId_v2, confirm the new path is healthy under production traffic, then stop writing the old key and retire its index.

python 
CONTRACT = """
import org.janusgraph.core.schema.SchemaAction
import org.janusgraph.core.schema.SchemaStatus
import org.janusgraph.graphdb.database.management.ManagementSystem
mgmt = graph.openManagement()
mgmt.updateIndex(mgmt.getGraphIndex('byUserId'), SchemaAction.DISABLE_INDEX).get()
mgmt.commit()
ManagementSystem.awaitGraphIndexStatus(graph, 'byUserId').status(SchemaStatus.DISABLED).call()
mgmt = graph.openManagement()
mgmt.updateIndex(mgmt.getGraphIndex('byUserId'), SchemaAction.REMOVE_INDEX).get()
mgmt.commit(); 'CONTRACTED'
"""
client = Client("ws://janusgraph-server:8182/gremlin", "g")
print(client.submit(CONTRACT).all().result())
client.close()

Disable before remove: DISABLE_INDEX takes the old index out of planning but leaves it re-enable-able, so the cutover stays reversible right up until REMOVE_INDEX reclaims its storage.

Verify: the old index is gone and reads resolve through the new key.

python 
count_new = g.V().has("userId_v2", 100073).count().next()
print(f"new-key lookup returned {count_new} vertex(es)")
# expect the same count the old-key lookup returned before cutover

Fallback and rollback procedures

Validate between actions rather than stacking them. Each step has a defined recovery path.

If Step 1 fails (partial or rejected expand). The try/rollback guard means a failed expand registers nothing, so the schema is untouched. Fix the script — a common cause is a dataType mismatch or a lock still held by a prior run — and re-run; the containsPropertyKey guard makes the retry safe.

If Step 2 shows incomplete dual-write. Do not backfill. A replica still on the old binary is writing only the old key, and backfilling now would leave new writes on that replica un-migrated. Finish rolling the dual-write deploy across the entire fleet, re-run the Step 2 verification, and only then proceed.

If Step 3 stalls or the index will not drain. A backfill whose writes never become queryable usually means the mixed index is stuck in REGISTERED or the dispatch queue is saturated. Pause the backfill, let the queue drain, and confirm index status; the recovery mechanics are in Reindexing & Recovery.

If Step 4 finds a stable mismatch. The transform is wrong — stop before cutover. Because reads are still on the old key, the graph is unharmed; correct the backfill logic, clear the bad userId_v2 values, and re-run Steps 3 and 4. The full reversal is in Rolling Back a Failed Schema Change.

If Step 5 regresses after cutover but before REMOVE_INDEX. Re-enable the old index and redeploy the old readers. Because you disabled rather than removed, the old query path is one ENABLE_INDEX away:

groovy 
mgmt = graph.openManagement()
mgmt.updateIndex(mgmt.getGraphIndex('byUserId'), SchemaAction.ENABLE_INDEX).get()
mgmt.commit()

Once REMOVE_INDEX has run, the old index is gone and rollback requires rebuilding it with a fresh REINDEX — which is why the contract step is the migration’s only true point of no return.

Frequently Asked Questions

Do I have to do two separate application deploys? Yes — one to enable dual-write and a later one to cut reads to the new key. Splitting them is deliberate: the gap between the deploys is the dual-shape window where the backfill runs and parity is proven while old readers still see valid data. Collapsing both into one deploy removes the safety margin and risks new readers querying a key the backfill has not yet populated everywhere.

How long should I stay in the dual-shape window? Long enough for the backfill to complete and parity to hold on a stable sample, plus a soak period under real traffic. There is no hard clock — the window is safe to hold for hours or days because both shapes are fully written and reads never left the old key. Do not shorten it to hit a deadline; the window existing is what makes the migration reversible.

What if a vertex’s old value cannot be converted to the new type? The backfill logs and skips it rather than aborting, so one malformed value never blocks the whole job. Those skipped rows are a data-quality finding: collect them from the logs, decide on a correction (a default, a manual fix, or exclusion), and handle them before cutover, because a read on the new key will miss any vertex whose new value was never written.

Can I run the backfill in parallel across multiple workers? Yes. The job selects only vertices lacking the new key and writes conditionally with coalesce, so two workers that happen to pick the same vertex converge on the same result instead of double-writing. Parallelism speeds up a large backfill; just size the driver connection pool to the worker count so the extra concurrency does not starve threads.