Rolling Back a Failed Schema Change

When a live schema migration goes wrong, the graph does not usually crash — it degrades quietly, with a subset of writes throwing SchemaViolationException, an index frozen short of ENABLED, or reads that just started missing rows nobody deleted. This procedure is the disciplined reversal: how to notice the failure early, freeze the rollout before it spreads, back the offending schema element out, and reconcile any data already written under the new shape so the graph returns to a coherent state. It sits under the Schema Migration Patterns reference and assumes you followed an expand-migrate-contract flow such as Performing Online Schema Migrations; the failure it addresses is the half-applied migration that is neither the clean old shape nor the finished new one. The governing fact that makes rollback tractable is that everything before the contract phase is reversible — the old shape is still written and still read — so the entire job is to get back to it without losing the writes that landed in between.

A migration timeline with the rollback branch that peels the new shape back off The top row shows the forward migration: expand the schema, dual-write, backfill, then a detection point. At the detection point a downward rollback branch splits off in amber. The rollback branch runs right to left through four steps: stop the rollout, disable or remove the offending index or element, restore dual-write to the old shape as the authority, and reconcile any data written under the new shape. The branch rejoins the old-shape baseline, showing that the graph is returned to the shape it had before the migration began. expand dual-write backfill detect failure errors · stuck · missing roll back 1 · stop the rollout 2 · disable / remove offending element 3 · restore dual-write to old shape 4 · reconcile new data back to coherent old shape reversible because reads never left the old key before contract
The rollback branch unwinds the migration in reverse and rejoins the old-shape baseline.

Prerequisites

Confirm these before you touch anything — a panicked rollback that skips the detection step often removes an index that was merely lagging, turning a recoverable stall into a real outage.

  • Confirmation you are pre-contract. Verify that reads still resolve through the old key and the old index is still ENABLED. If REMOVE_INDEX has already run, this is a rebuild, not a rollback — go to Reindexing & Recovery instead.
  • Management API access through the Gremlin console or a service account, plus gremlinpython matching your server’s TinkerPop line for the reconciliation job.
  • The original migration’s transform, so you can compute which vertices were touched and reverse or reconcile them deterministically.
  • JMX or Prometheus visibility into index status and SchemaViolationException counts, and a rollout tool that can be paused mid-deploy.
  • Cluster health. nodetool status shows UN for all storage nodes; a rollback against a partitioned cluster can double-apply and should wait for the partition to heal.

Step 1 — Detect the bad migration and stop the rollout

Rollback begins with a correct diagnosis, because the three failure signatures each demand a different response. Distinguish them before acting:

python 
# Signature A: writes rejected — the new key's type or registration is wrong.
viol = int(client.submit(
    "g.getOpenTransactions().size()"      # sanity; then scrape the JMX counter
).all().result()[0])
# Scrape SchemaViolationException rate from the StandardJanusGraph bean.

# Signature B: index frozen — never advanced past REGISTERED.
status = client.submit(
    "graph.openManagement().getGraphIndex('byUserIdV2')"
    ".getIndexStatus(graph.openManagement().getPropertyKey('userId_v2')).toString()"
).all().result()[0]
print("new index status:", status)   # REGISTERED here means stuck

# Signature C: reads missing rows — parity broke after a premature cutover.
missing = client.submit(
    "g.V().has('userId').not(__.has('userId_v2')).count()"
).all().result()[0]
print("vertices missing new key:", missing)

The moment any signature confirms a genuine failure — not transient lag — freeze the rollout so no further replicas adopt the bad shape.

bash 
# Halt the in-flight application deploy (example: Kubernetes rollout).
kubectl rollout pause deployment/ingestion-service

Verify: the rollout is paused and the failure signal has stopped climbing.

bash 
kubectl rollout status deployment/ingestion-service | grep -i paused
# then confirm the SchemaViolationException rate has flattened, not risen

Step 2 — Disable or remove the offending element

Take the bad element out of the query path. A stuck or misdefined index should be disabled first — DISABLE_INDEX is instantly reversible and stops the planner from routing to it — and only removed if you are certain it must be rebuilt from scratch.

python 
ROLLBACK_INDEX = """
import org.janusgraph.core.schema.SchemaAction
import org.janusgraph.core.schema.SchemaStatus
import org.janusgraph.graphdb.database.management.ManagementSystem
mgmt = graph.openManagement()
idx = mgmt.getGraphIndex('byUserIdV2')
mgmt.updateIndex(idx, SchemaAction.DISABLE_INDEX).get()
mgmt.commit()
ManagementSystem.awaitGraphIndexStatus(graph, 'byUserIdV2')
              .status(SchemaStatus.DISABLED).call().getSucceeded()
"""
client = Client("ws://janusgraph-server:8182/gremlin", "g")
print(client.submit(ROLLBACK_INDEX).all().result())
client.close()

If the index was only lagging, do not remove it — disabling already stopped the bleeding, and you can re-enable it once the underlying cause is fixed. Remove it only when the definition itself is wrong:

groovy 
mgmt = graph.openManagement()
mgmt.updateIndex(mgmt.getGraphIndex('byUserIdV2'), SchemaAction.REMOVE_INDEX).get()
mgmt.commit()

The new property key itself cannot be dropped while any vertex carries it, so leave it registered and relabel it as abandoned; the reconciliation in Step 4 clears its values.

groovy 
mgmt = graph.openManagement()
mgmt.changeName(mgmt.getPropertyKey('userId_v2'), 'userId_v2_abandoned')
mgmt.commit()

Verify: the offending index is DISABLED (or absent) and no longer used by the planner.

groovy 
mgmt = graph.openManagement()
println mgmt.containsGraphIndex('byUserIdV2') ?
    mgmt.getGraphIndex('byUserIdV2').getIndexStatus(mgmt.getPropertyKey('userId_v2_abandoned')) :
    'REMOVED'
// expect: DISABLED or REMOVED

Step 3 — Restore dual-write to the old shape

Redeploy application code that treats the old key as the sole authority again. If you were pre-cutover, this is just reverting the dual-write deploy to old-key-only; if a premature cutover already moved reads to the new key, this step also moves reads back — the old key is still fully populated because the migration never stopped writing it.

python 
def write_account_rolledback(g, user_id_str: str, region: str):
    # Authority is the old key again. The new key is no longer written or read.
    (
        g.V().has("userId", user_id_str).fold()
         .coalesce(__.unfold(), __.addV("account").property("userId", user_id_str))
         .property("region", region)
         .iterate()
    )

Roll this to the entire fleet and resume the (now-reverting) deploy you paused in Step 1.

Verify: reads resolve through the old key and no replica still writes the new one.

python 
# Old-key lookup returns the expected vertex.
assert g.V().has("userId", "100073").hasNext(), "old-key read path broken"
# No new writes are landing on the abandoned key.
before = g.V().has("userId_v2_abandoned").count().next()
# ... wait one write cycle ...
after = g.V().has("userId_v2_abandoned").count().next()
assert after <= before, "a replica is still writing the new key"

Step 4 — Reconcile data written under the new shape

Any vertex that only ever received the new key — created by a new binary during the window, never carrying the old key — is now orphaned from the old-shape query path and must be reconciled. Back-fill the old key from the new value, then clear the abandoned key so no trace of the failed shape remains.

python 
from gremlin_python.process.graph_traversal import __

def reconcile(g, batch_size=500):
    fixed = 0
    while True:
        # Vertices that have the new key but are missing the old one:
        # the reverse of the original backfill's selection.
        batch = (
            g.V().has("userId_v2_abandoned").not_(__.has("userId"))
             .limit(batch_size)
             .project("vid", "new").by(__.id_()).by("userId_v2_abandoned")
             .toList()
        )
        if not batch:
            break
        for row in batch:
            old_val = str(row["new"])            # reverse the retype: Long -> String
            (
                g.V(row["vid"])
                 .coalesce(__.has("userId"), __.property("userId", old_val))
                 .iterate()
            )
            fixed += 1
        print(f"reconciled: {fixed}")
    # Then strip the abandoned key everywhere so the old shape is clean.
    g.V().has("userId_v2_abandoned").properties("userId_v2_abandoned").drop().iterate()
    return fixed

Verify: every vertex carries the old key and none carries the abandoned key.

python 
orphans = g.V().has("userId_v2_abandoned").not_(__.has("userId")).count().next()
residue = g.V().has("userId_v2_abandoned").count().next()
print(f"orphans={orphans} residue={residue}")
# expect both 0 — the graph is back to a coherent old shape

Fallback and rollback procedures

Because this is the rollback, the fallbacks cover a rollback that itself stalls.

If Step 1 cannot distinguish a stall from a failure. Treat a REGISTERED index as lag, not failure, for the first poll interval — read the wall-clock time in REGISTERED and only declare failure if it plateaus. Removing an index that was about to enable turns a self-healing lag into a rebuild.

If Step 2’s DISABLE_INDEX will not reach DISABLED. An unreachable JanusGraph instance blocks the status transition. Confirm every node is up; if one is dead, clear its stale registration with mgmt.forceCloseInstance(...) and re-issue the disable. Do not force REMOVE_INDEX past a blocked disable — that can strand index data.

If Step 3 finds the old key was not fully written. This means the failure happened after a premature cutover and dual-write had been switched off — the dangerous case. Stop writes entirely, run the Step 4 reconciliation first to rebuild the old key from the new one, and only then restore old-key reads, or you will serve incomplete results during the reconcile.

If Step 4 reconciliation is interrupted. It is idempotent by the same absence-guard as the forward backfill, so resume it; the coalesce guard means already-reconciled vertices are skipped. Do not run the abandoned-key drop() until the reconcile loop reports zero orphans, or you will destroy the only copy of a value that was never mirrored to the old key.

Wire every one of these signals — the SchemaViolationException spike, the stuck REGISTERED index, the missing-row count — into the on-call path so a failing migration pages a human immediately rather than surfacing as user-visible latency; the severity classification and routing are covered in Alert Routing for Violations.

Frequently Asked Questions

How do I tell a failed migration from an index that is just lagging? Read the wall-clock time the new index has spent in REGISTERED. Lag advances — the status moves toward ENABLED and the missing-row count shrinks on re-check. A genuine failure plateaus: the index sits in REGISTERED indefinitely, or SchemaViolationException keeps climbing. Give it one poll interval before declaring failure, because disabling or removing an index that was about to enable converts a self-healing stall into a full rebuild.

Can I drop the new property key to undo the migration? No. JanusGraph will not drop a property key while any vertex still carries it, and the migration wrote that key onto real data. Instead, relabel it as abandoned with changeName, reconcile its values back onto the old key for any vertex that lacks the old shape, then drop the property values everywhere with a traversal. The key registration remains, inert, but the data is clean.

What if reads were already cut over to the new key when the failure hit? Then the rollback also has to move reads back to the old key, which is safe only because the migration never stopped writing the old key before the contract phase. Restore old-key reads via redeploy, and if any vertices were created new-key-only during the window, run the reconciliation first so the old-key read path is complete before you trust it.

Is the reconciliation job safe to re-run after an interruption? Yes. It selects only vertices that have the new key but lack the old one and writes conditionally with coalesce, so already-reconciled vertices are skipped and a resumed run converges. The one ordering rule is to never run the final abandoned-key drop() until the orphan count reads zero, because that drop destroys the only copy of any value not yet mirrored back to the old key.