ScyllaDB Migration Rollback Procedure

The moment a ScyllaDB cutover breaches its latency or error SLO, the clock that matters is not “how fast can we fix ScyllaDB” but “how fast can we get back to Cassandra without losing the writes that already landed on Scylla.” This runbook sits under the ScyllaDB Migration reference and covers only the reverse path: the pre-cutover checkpoint that makes rollback possible, the signals that trip the decision gate, the exact sequence to reverse traffic to the retained Cassandra cluster, and — the step most rollback plans omit — reconciling the writes that committed to ScyllaDB during the cutover window so Cassandra does not silently drift behind. The specific failure it prevents is the irreversible one: a team that repoints traffic back to Cassandra, declares the incident closed, and discovers a week later that every vertex created during the 40-minute cutover window exists only on a ScyllaDB cluster they have since torn down.

The cutover timeline with its rollback branch at the decision gate A left-to-right timeline: a pre-cutover checkpoint, then a staged cutover ramping traffic onto ScyllaDB, reaching a decision-gate diamond. If the SLO stays green the timeline continues up to a full one-hundred-percent cutover. If the SLO breaches on latency or error rate, the timeline branches downward into the rollback lane: reverse traffic to the retained Cassandra cluster, reconcile the writes that landed on ScyllaDB during the window, verify index parity with a reindex, and finish with a post-mortem. The reconcile step is highlighted because skipping it silently loses every write committed during the cutover window. Pre-cutover checkpoint snapshot + retain C* Staged cutover 10% → 50% watch SLO decision gate SLO green → proceed Full cutover 100%, C* on standby SLO breach → rollback Reverse traffic to Cassandra repoint router Reconcile window writes replay Scylla → C* Verify index parity (reindex) delta → 0 Post-mortem root-cause before retry The reconcile step is non-optional: skipping it strands every write committed to ScyllaDB during the cutover window.

Prerequisites

Confirm every item before cutover begins — a rollback plan assembled after the SLO breaches is already too late. The single non-negotiable is a Cassandra cluster kept alive and write-consistent up to the cutover instant.

  • A retained Cassandra cluster, not decommissioned, with its janusgraph.properties preserved. The source-side Replication Strategies must still be intact so reversed traffic lands on the same NetworkTopologyStrategy topology it left.
  • A cutover-instant checkpoint — either a dual-write path mirroring mutations to both clusters, or a nodetool snapshot on Cassandra taken at the moment traffic shifts, so reconciliation has a known boundary.
  • A traffic router you can flip — a service-discovery entry, load-balancer target, or the storage.hostname list itself — that repoints JanusGraph’s storage layer between clusters without a rebuild.
  • Defined SLO thresholds captured as concrete numbers before cutover: P99 traversal latency and CQL error rate, each with a breach value and a sustain duration.
  • cqlsh, nodetool, and gremlin-python access to both clusters, plus permission to run a mixed-index reindex through the JanusGraph Management API for the parity check in Step 6.
  • The applied Cassandra vs ScyllaDB Configuration Diff on hand, so reversing to Cassandra means restoring a known property set rather than reconstructing it under pressure.

Step 1 — Establish the pre-cutover checkpoint

Before any traffic shifts, freeze a recoverable boundary on Cassandra. A snapshot is the minimum; a dual-write window is stronger because it eliminates the reconciliation step entirely. Take the snapshot on every Cassandra node and record the timestamp — that timestamp is the lower bound of the reconciliation window in Step 5.

bash 
# On every Cassandra node — tag the snapshot with the cutover marker
nodetool snapshot -t precutover_$(date +%Y%m%dT%H%M%S) janusgraph_prod

# Record the exact cutover-start epoch for the reconciliation window bound
date +%s > /var/run/janusgraph/cutover_start.epoch

If you run a dual-write path instead, confirm it is mirroring to both clusters before proceeding — a dual-write path that has silently dropped one leg gives false confidence.

Verify: the snapshot exists on every node and the cutover marker is recorded.

bash 
nodetool listsnapshots | grep precutover_    # must appear on each Cassandra node

Step 2 — Detect the regression

Rollback is triggered by the SLO, not by intuition. Watch two series against the thresholds fixed in the prerequisites: P99 traversal latency and CQL error rate. A single spike is noise; a sustained breach past the agreed duration is the trigger.

bash 
# P99 traversal latency against baseline (JMX → Prometheus)
curl -s http://localhost:9090/api/v1/query --data-urlencode \
  'query=histogram_quantile(0.99, rate(gremlin_traversal_ms_bucket[5m]))' \
  | jq '.data.result[0].value[1]'

# CQL error rate on the ScyllaDB path
curl -s http://localhost:9090/api/v1/query --data-urlencode \
  'query=rate(cql_request_errors_total[5m])' | jq '.data.result'

Verify: the breach is sustained, not transient — confirm the metric stays over threshold across at least three consecutive scrape windows before declaring a regression, and cross-check scylla_compaction_manager_pending is not simply a compaction backlog that will drain on its own.

Step 3 — Pass the decision gate

The decision gate is an explicit, pre-agreed rule, not a debate held during the incident. Reverse traffic if either condition holds for the sustain duration: P99 latency exceeds baseline by more than 20%, or the CQL error rate exceeds the agreed ceiling. Below those, hold and investigate; at or above, roll back — do not attempt to fix ScyllaDB with live production traffic on it.

groovy 
// Decision-gate check run from the Gremlin console or an ops script
p99Breach   = currentP99Ms > baselineP99Ms * 1.20
errorBreach  = cqlErrorRate > errorRateCeiling
rollback     = (p99Breach || errorBreach) && sustainedFor(minutes: 5)
println "rollback decision = ${rollback}"

Verify: the gate decision is logged with the metric values that triggered it, so the post-mortem in Step 7 works from data rather than recollection.

Step 4 — Reverse traffic to Cassandra

Repoint the storage layer back to the retained Cassandra cluster and restore its property set. Reverse in the same staged fashion you cut over — ramp reads back first, then writes — so a half-reverted state never has writes going to one cluster and reads from the other.

bash 
# Restore the Cassandra janusgraph.properties (retained from before cutover)
cp /etc/janusgraph/janusgraph.properties.cassandra \
   /etc/janusgraph/janusgraph.properties

# Roll the Gremlin Server pools back onto Cassandra, draining each first
for host in gremlin-01 gremlin-02 gremlin-03; do
  ssh "$host" "gremlin -e 'graph.close()' && systemctl restart janusgraph"
done

Verify: confirm live traffic is served from Cassandra and the ScyllaDB path is quiescent.

cql 
-- Run against Cassandra: coordinator should now be a Cassandra node
SELECT coordinator, request FROM system_traces.sessions LIMIT 1;
bash 
# ScyllaDB request rate must fall to zero as traffic drains off it
curl -s http://scylla-node-01:9180/metrics | grep -E "scylla_transport_requests_served"

Step 5 — Reconcile writes stranded on ScyllaDB

This is the step that makes rollback safe rather than lossy. Every mutation committed to ScyllaDB between the checkpoint epoch and traffic reversal exists only on ScyllaDB. Extract those vertices and edges and replay them into Cassandra before you consider the rollback complete. Scope the extraction by the write-time window recorded in Step 1.

python 
from gremlin_python.driver.driver_remote_connection import DriverRemoteConnection
from gremlin_python.process.anonymous_traversal import traversal
from gremlin_python.process.traversal import P

# Source = ScyllaDB (still up), sink = Cassandra (now serving)
scylla = traversal().with_remote(
    DriverRemoteConnection("ws://scylla-gremlin:8182/gremlin", "g"))
cassandra = traversal().with_remote(
    DriverRemoteConnection("ws://cassandra-gremlin:8182/gremlin", "g"))

CUTOVER_START_MS = 1_752_000_000_000  # from cutover_start.epoch * 1000

# Vertices written on ScyllaDB during the window carry a createdAt property
stranded = (scylla.V()
            .has("createdAt", P.gte(CUTOVER_START_MS))
            .value_map(True).to_list())

replayed = 0
for vp in stranded:
    label = vp["label"]
    props = {k: v[0] for k, v in vp.items() if k not in ("id", "label")}
    t = cassandra.add_v(label)
    for k, v in props.items():
        t = t.property(k, v)
    t.iterate()                        # idempotent on the natural key
    replayed += 1
print(f"replayed {replayed} stranded vertices into Cassandra")

scylla.close()
cassandra.close()

For a dual-write deployment, this step reduces to confirming the dual-write leg never gapped — query both clusters for the window’s vertex count and assert equality instead of replaying.

Verify: the window vertex counts match across both clusters after replay.

python 
scylla_ct    = scylla.V().has("createdAt", P.gte(CUTOVER_START_MS)).count().next()
cassandra_ct = cassandra.V().has("createdAt", P.gte(CUTOVER_START_MS)).count().next()
assert scylla_ct == cassandra_ct, f"drift: scylla={scylla_ct} cassandra={cassandra_ct}"

Step 6 — Verify index parity via reindex

Storage reconciliation is not index reconciliation. Replaying vertices into Cassandra writes storage and composite indexes, but the mixed (Elasticsearch/OpenSearch) index must be rebuilt so has() predicates resolve the replayed data. Trigger a targeted reindex through the Management API and confirm the document-count delta closes to zero — the same recovery mechanics detailed in Reindexing & Recovery.

groovy 
// Gremlin console against the Cassandra-backed graph
mgmt = graph.openManagement()
idx  = mgmt.getGraphIndex("vertexByName")
mgmt.updateIndex(idx, SchemaAction.REINDEX).get()
mgmt.commit()
ManagementSystem.awaitGraphIndexStatus(graph, "vertexByName")
  .status(SchemaStatus.ENABLED).call()

Verify: the mixed-index document count matches the storage vertex count for the reindexed label.

bash 
# Search-tier doc count should equal the storage count once reindex completes
curl -s "http://opensearch:9200/janusgraph_vertexbyname/_count" | jq '.count'

Step 7 — Post-mortem before any retry

The rollback is not complete when traffic is stable; it is complete when the regression’s root cause is understood. Pull the decision-gate log from Step 3, the ScyllaDB metric that breached, and the compaction and shard-utilization series from the cutover window. Do not schedule a second cutover attempt until the root cause maps to a specific configuration or capacity change — a retry against an un-diagnosed regression reproduces it.

Verify: the post-mortem names a concrete cause (for example a local-datacenter mismatch, an under-sized pool, or a compaction backlog) and a corresponding fix in the Cassandra vs ScyllaDB Configuration Diff before the next attempt is booked.

Fallback and rollback procedures

Each step has a failure mode with a defined recovery; resolve it before moving on.

If Step 1’s snapshot is incomplete. A snapshot missing from any node leaves that node’s range unrecoverable. Re-run nodetool snapshot fleet-wide and confirm on every node before cutover — never begin cutover on a partial checkpoint.

If Step 2 cannot distinguish a regression from a compaction backlog. Hold at the current traffic share and watch scylla_compaction_manager_pending. A draining backlog recovers on its own; a flat or growing one that also breaches latency is a genuine regression — proceed to the gate.

If Step 4’s reversal leaves a split-brain (reads on one cluster, writes on the other). Halt the ramp immediately and force all traffic to Cassandra in one move rather than continuing a staged reverse. A split serving path corrupts read-your-writes for the duration; a brief full-Cassandra cutover is safer than a prolonged split.

If Step 5 replay fails or the counts do not match. Do not tear down ScyllaDB. Keep it up as the authoritative source for the window until the replay reconciles and the counts match — ScyllaDB is the only copy of those writes. Retry the replay scoped to the missing key range, and if a natural-key collision blocks an upsert, resolve it manually before continuing.

If Step 6’s reindex stalls. Serve reads from storage-backed traversals (g.V().has('name','x')) while the mixed index rebuilds, exactly as the async-index recovery path in Reindexing & Recovery prescribes, rather than exposing a half-built search index to production reads.