Cassandra vs ScyllaDB Configuration Diff

Repointing storage.hostname at a ScyllaDB cluster leaves most of janusgraph.properties untouched, and that is precisely the trap: the handful of keys that should change are the ones that decide whether the shard-aware driver routes each mutation to its token-owning core or scatters it across every shard. This guide is the line-by-line configuration diff under the ScyllaDB Migration reference — it isolates exactly which properties differ between a Cassandra deployment and a ScyllaDB one, which must stay byte-for-byte identical so a latency regression is never ambiguous, and why each delta exists at the driver, pool, consistency, and compaction layers. The failure it prevents is the silent one: a config that connects, passes a smoke test, and then doubles cross-shard hops on the first ingestion burst because max-requests-per-connection was left at the Cassandra default and compaction was never re-declared.

The four configuration lanes that change moving JanusGraph from Cassandra to ScyllaDB Four stacked lanes — driver routing, connection pool, consistency, and compaction — each showing the Cassandra value on the left, an arrow, and the ScyllaDB value on the right, with a short reason column. The driver lane changes from token-aware routing to shard-aware routing keyed to local-datacenter. The pool lane raises max-requests-per-connection from 1024 to 4096 and lowers physical sockets toward per-core alignment. The consistency lane is unchanged by design, keeping LOCAL_QUORUM on both reads and writes so a regression is attributable. The compaction lane changes from SizeTieredCompactionStrategy to ScyllaDB's IncrementalCompactionStrategy declared on the table. CASSANDRA ScyllaDB WHY Driver routing token-aware routing local-datacenter set shard-aware routing local-datacenter pinned per core route to token-owning shard, no internal hop Pool sockets requests/conn = 1024 max-conns/host = 12 requests/conn = 4096 max-conns/host = 8 (per core) shard reactor multiplexes more frames per socket Consistency LOCAL_QUORUM r/w system-ops local = true LOCAL_QUORUM r/w unchanged by design hold constant so a regression is attributable Compaction table-level SizeTieredCompaction STCS default IncrementalCompaction ICS, re-declared on table bounded space overhead, steadier read latency Solid arrow = value changes · dashed arrow = value must stay identical across the switch

Prerequisites

Confirm every item before you diff a single key. The most common cause of a “clean” config diff that regresses is treating a table-level compaction change as if it were a janusgraph.properties change — the two are applied through different surfaces and validated differently.

  • A warm ScyllaDB target. ScyllaDB Open Source 5.x or Enterprise 2024.x with native_transport_port 9042 reachable from every JanusGraph node, and the keyspace already re-created with NetworkTopologyStrategy per your Replication Strategies. Auto-creation defaults to SimpleStrategy and will silently invalidate the routing deltas below.
  • The live Cassandra janusgraph.properties under version control, so the diff is against a known baseline rather than an operator’s memory of it.
  • DataStax Java Driver 4.x (bundled with the JanusGraph CQL adapter). The property names here are the JanusGraph-namespaced storage.cql.* keys, not raw driver keys.
  • cqlsh access to both clusters plus permission to run ALTER TABLE ... WITH compaction on the ScyllaDB keyspace.
  • A captured P95/P99 latency baseline from the Cassandra cluster via nodetool cfstats, so each applied delta is measured, not assumed. Keep the Cassandra cluster reachable as a rollback target — the ScyllaDB Migration Rollback Procedure depends on it existing.

Step 1 — Audit the current Cassandra configuration

Extract the exact storage.cql.* surface from the running Cassandra config and the table-level compaction strategy from storage. You cannot diff what you have not enumerated, and JanusGraph’s defaults differ by version, so read the live values rather than the documentation defaults.

bash 
# Pull the storage.cql.* surface from the live properties file
grep -E "^storage\.(backend|hostname|cql\.)" /etc/janusgraph/janusgraph.properties \
  | sort

# Read the table-level compaction strategy actually in effect on Cassandra
cqlsh -e "SELECT table_name, compaction FROM system_schema.tables \
  WHERE keyspace_name = 'janusgraph_prod';"

Verify: the audit must list read-consistency-level, write-consistency-level, max-requests-per-connection, max-connections-per-host, core-connections-per-host, and local-datacenter. If any is absent, JanusGraph is running the driver default for it — record the default explicitly so the diff is complete.

bash 
grep -Ec "max-requests-per-connection|max-connections-per-host|local-datacenter" \
  /etc/janusgraph/janusgraph.properties   # expect 3; a lower count means defaults are in play

Step 2 — Produce the property diff

With the baseline enumerated, apply the deltas below. The table is the authoritative diff — every key that changes, every key that must not, and the reason. Values assume a three-node local datacenter with 16-vCPU ScyllaDB nodes; scale the pool row to your core count.

janusgraph.properties key Cassandra value ScyllaDB value Why it changes
storage.cql.local-datacenter us-east-1 us-east-1 (required) Same key, higher stakes: an unset or wrong DC name drops shard-aware routing back to round-robin and doubles cross-shard hops per mutation
storage.cql.max-requests-per-connection 1024 4096 ScyllaDB’s shard-per-core reactor multiplexes far more async frames per socket; the Cassandra value under-utilizes each connection and forces extra sockets
storage.cql.max-connections-per-host 12 8 Align physical sockets toward one-per-core rather than Cassandra’s few-connections-per-node model, so the shard-aware driver routes without an internal hop
storage.cql.core-connections-per-host 4 8 Keep a warm per-core floor so a cutover traffic spike does not pay cold-start latency on newly opened sockets
storage.cql.read-consistency-level LOCAL_QUORUM LOCAL_QUORUM Unchanged by design — moving the acknowledgment boundary and the backend at once makes every latency regression ambiguous
storage.cql.write-consistency-level LOCAL_QUORUM LOCAL_QUORUM Unchanged by design — same durability boundary carried across the switch
storage.cql.only-use-local-consistency-for-system-operations true true Unchanged — keeps ID allocation and schema locks on LOCAL_QUORUM instead of escalating to cross-DC QUORUM while the storage cluster warms
storage.cql.request-timeout 12000 15000 Widen slightly above worst-case incremental-compaction or index-flush duration so initial-load WriteTimeout storms do not mask the real backlog
driver retry policy DowngradingConsistencyRetryPolicy (removed) — app-level backoff Deprecated in driver 4.x, removed in 4.14; replace with application-level exponential backoff so consistency degradation is explicit

The consistency rows are in the table specifically to be marked “do not touch.” The single most damaging shortcut in a backend switch is to re-tune durability at the same moment you change the backend; when P99 moves afterward you have no way to attribute it. Keep the acknowledgment boundary identical — the reasoning is worked through in Optimizing ScyllaDB Read/Write Consistency for Graphs.

The resulting ScyllaDB janusgraph.properties block, annotated:

properties 
storage.backend=cql
storage.hostname=scylla-node-01,scylla-node-02,scylla-node-03
storage.port=9042
storage.cql.keyspace=janusgraph_prod

# CHANGED: required for shard-aware routing — must match nodetool status exactly
storage.cql.local-datacenter=us-east-1

# CHANGED: raise frame multiplexing to match the shard-per-core reactor
storage.cql.max-requests-per-connection=4096
# CHANGED: align physical sockets toward per-core, keep a warm floor
storage.cql.max-connections-per-host=8
storage.cql.core-connections-per-host=8
# CHANGED: widen above worst-case incremental compaction duration
storage.cql.request-timeout=15000

# UNCHANGED: identical acknowledgment boundary across the switch
storage.cql.read-consistency-level=LOCAL_QUORUM
storage.cql.write-consistency-level=LOCAL_QUORUM
storage.cql.only-use-local-consistency-for-system-operations=true

# Load-phase only; MUST be false on the serving path
storage.batch-loading=false

Compaction is not a janusgraph.properties key — it is declared on the table. Re-declare ScyllaDB’s IncrementalCompactionStrategy on the JanusGraph storage tables so space amplification stays bounded and read latency stays steady under the write-heavy edge mutations a graph produces:

cql 
-- Run on the ScyllaDB cluster, per JanusGraph storage table (edgestore, graphindex, ...)
ALTER TABLE janusgraph_prod.edgestore
  WITH compaction = {'class': 'IncrementalCompactionStrategy'};

ALTER TABLE janusgraph_prod.graphindex
  WITH compaction = {'class': 'IncrementalCompactionStrategy'};

Verify: diff the generated ScyllaDB file against the Cassandra baseline and confirm only the intended keys moved.

bash 
diff <(sort /etc/janusgraph/janusgraph.properties.cassandra) \
     <(sort /etc/janusgraph/janusgraph.properties.scylla)
# Expect changes ONLY on local-datacenter, max-requests-per-connection,
# max/core-connections-per-host, request-timeout. Any consistency-level line
# in the diff is a mistake — revert it.

Step 3 — Apply the diff on a canary node

Never roll the diff to the fleet in one move. Apply the ScyllaDB janusgraph.properties to a single canary Gremlin Server pointed at the ScyllaDB cluster, restart it, and confirm the driver attaches with the new pool rather than falling back to defaults on a malformed key.

bash 
# On the canary only
cp /etc/janusgraph/janusgraph.properties.scylla \
   /etc/janusgraph/janusgraph.properties
systemctl restart janusgraph

# Confirm the shard-aware pool and datacenter attached, not a default fallback
grep -E "local-datacenter|core-connections|max-requests|shard" \
  /var/log/janusgraph/server.log | tail -8

Verify: trace a single write on the canary and read the consistency level and shard routing back from ScyllaDB, proving the applied config is the config in effect.

cql 
-- Confirm LOCAL_QUORUM is what the coordinator actually applied (not a stale default)
SELECT consistency_level, coordinator FROM system_traces.sessions
  WHERE session_id = <trace_id>;

Step 4 — Validate under representative load

A config that attaches cleanly can still regress under concurrency. Drive ~1.5x production write rate at the canary and watch the three series the diff targets: shard reactor_utilization, driver pool in-flight against the new ceiling, and P99 traversal latency against the captured baseline.

bash 
# Shard hotspotting — a wrong local-datacenter shows here as uneven reactor load
curl -s http://scylla-node-01:9180/metrics | grep -E "reactor_utilization"

# Driver pool saturation against the new max-connections-per-host=8
curl -s http://localhost:9090/api/v1/query \
  --data-urlencode 'query=cql_pool_in_flight' | jq '.data.result'

Verify: P99 must stay within your SLO and reactor_utilization must be even across shards — uneven load is the signature of a routing regression, almost always a local-datacenter mismatch. Promote the diff to the fleet via configuration management only after a 30-minute clean canary window, restarting pools in a rolling fashion and confirming every node reports identical max-requests-per-connection and max-connections-per-host.

Fallback and rollback procedures

Validate between actions rather than stacking deltas; each has a defined recovery path.

If Step 1 shows missing keys. The driver is running defaults you did not record. Add the default value explicitly to the Cassandra baseline before diffing, or the ScyllaDB config will appear to change a key that was never set and the diff review will misfire.

If Step 2’s diff touches a consistency line. Revert that line immediately. The consistency rows exist to stay constant; a consistency change riding along with the backend switch is the top cause of an unattributable post-cutover regression.

If Step 3 falls back to driver defaults. A malformed property or wrong local-datacenter makes the driver ignore the block. Match the datacenter name to nodetool status output exactly, fix the typo, and restart the canary before continuing — do not proceed to load validation on a defaulted pool.

If Step 4 shows uneven shard utilization or P99 regression. Revert the canary’s janusgraph.properties to the Cassandra baseline, restart, and re-check local-datacenter and the partitioner match. If the regression is a compaction backlog rather than routing (scylla_compaction_manager_pending growing), let compaction drain before retrying rather than widening the pool. If load has already shifted and the regression breaches SLO, execute the full ScyllaDB Migration Rollback Procedure to reverse traffic to the retained Cassandra cluster.