Choosing Consistency Levels for Graph Reads

Picking a Cassandra or ScyllaDB consistency level for JanusGraph is not a single cluster-wide dial — it is a per-query-class decision, and this guide is the procedure for sorting your traversals into classes, mapping each to the weakest level that still meets its correctness bar, and proving the choice holds. It sits under the Eventual vs Strong Consistency reference and takes that page’s storage-versus-index distinction down to the storage tier alone: given a replication factor, which of ONE, LOCAL_ONE, LOCAL_QUORUM, QUORUM, or ALL does each read and write actually need. The failure this prevents is the two-sided mistake — a global QUORUM that taxes every analytical scan with cross-replica latency it does not need, or a global LOCAL_ONE that lets a recency-critical fraud lookup read a stale replica and make the wrong call. You classify once, map deterministically, and validate before you trust it.

Mapping graph query classes to CQL consistency levels under the R plus W greater than N overlap rule Query classes sort into two bands. Tolerant reads such as analytical scans and cache-warming route to ONE or LOCAL_ONE for lowest latency. Recency-critical reads such as fraud checks and read-after-write route to LOCAL_QUORUM or QUORUM. The center rule shows that when the read level R plus the write level W exceeds the replication factor N, the read and write replica sets are guaranteed to overlap, so the read sees the latest committed write. ALL sits at the top as maximum consistency at the cost of availability, used only for rare correctness-absolute writes. TOLERANT READS analytical scans cache warming · dashboards bulk backfill reads ONE / LOCAL_ONE RECENCY-CRITICAL READS fraud / authz checks read-after-write paths balance / inventory reads LOCAL_QUORUM / QUORUM the overlap rule R + W > N read + write sets overlap → read sees latest write ALL — maximum consistency one replica down = write fails · use sparingly worked example · N = 3 W=ONE (1) + R=ONE (1) = 2 ≤ 3 no overlap — read may be stale W=QUORUM (2) + R=QUORUM (2) = 4 > 3 overlap guaranteed — read is current pick the weakest pair whose R + W still clears N for the class that needs it
Sort each query into a band, then apply R + W > N: when the read and write replica counts sum past the replication factor, their sets overlap and the read is guaranteed current.

Prerequisites

Consistency choices are only meaningful against a known replication factor, so pin that down before you classify anything.

  • JanusGraph 0.6.x or 1.0.x on a CQL backend (Cassandra 3.11+/4.x or ScyllaDB), with storage.cql.read-consistency-level and storage.cql.write-consistency-level currently set to a known value.
  • A confirmed replication factor per keyspace. Run DESCRIBE KEYSPACE janusgraph_prod and read the replication map — every level below is relative to that N. If the topology spans datacenters, settle the Replication Strategies first, because LOCAL_QUORUM counts replicas in the local datacenter only.
  • An inventory of query classes. List the distinct read paths your application issues — you cannot map what you have not enumerated.
  • cqlsh and Gremlin console access, plus a staging cluster where you can drop a node to validate availability under each level without risking production.
  • The index boundary already decided. This guide tunes storage consistency; index visibility (bulk-refresh) is a separate lever covered by the parent reference, and conflating the two is the classic error.

Step 1 — Classify every query class as recency-critical or tolerant

Sort each read path into one of two buckets by a single test: if the query reads data a user or system just wrote and a stale answer causes an incorrect decision, it is recency-critical; otherwise it is tolerant. Write the classification down per class, not per query.

text 
Query class                     | Reads own recent write? | Wrong answer cost | Class
--------------------------------|-------------------------|-------------------|------------
Fraud-score vertex lookup       | yes                     | high (approve bad)| recency-critical
Post-signup profile read        | yes                     | medium (confusing)| recency-critical
Recommendation fan-out scan     | no                      | low (slightly old)| tolerant
Analytics / degree-distribution | no                      | none              | tolerant
Cache-warm bulk read            | no                      | none              | tolerant

The point of the table is to make the cost explicit. Most classes are tolerant, and forcing them all to QUORUM buys correctness nobody asked for while paying latency everybody feels.

Verify: every read path in your inventory has exactly one class and a one-line justification for it.

bash 
# A quick audit: grep your traversal code for read entry points and tag each
grep -rnE "g\.V\(\)|g\.E\(\)" src/ | wc -l   # sanity that the inventory count matches reality

Step 2 — Map each class to a level with the R + W > N rule

For recency-critical classes you need the read and write replica sets to overlap so the read cannot miss the latest write. With a coordinator counting replicas, that overlap is guaranteed exactly when:

R+W>NR + W > N

where N is the replication factor, W the replicas a write waits for, and R the replicas a read waits for. QUORUM on both sides means each waits for ⌊N/2⌋+1\lfloor N/2 \rfloor + 1 replicas, and for N = 3 that gives 2+2=4>32 + 2 = 4 > 3 — overlap holds. Tolerant classes do not need the inequality; they take the cheapest level.

properties 
# janusgraph.properties — set the default to the recency-critical requirement,
# because it is safer to widen the exception than to miss one.
storage.cql.read-consistency-level=LOCAL_QUORUM
storage.cql.write-consistency-level=LOCAL_QUORUM

Where the driver lets you scope per operation, override the tolerant read paths down to LOCAL_ONE; where it does not, route tolerant analytical work through a separate graph instance configured for the cheaper level:

properties 
# A second janusgraph.properties for the analytics/backfill instance only
storage.cql.read-consistency-level=LOCAL_ONE
storage.cql.write-consistency-level=LOCAL_QUORUM

The mapping in one line per band: tolerant reads take LOCAL_ONE (or ONE single-DC), recency-critical reads take LOCAL_QUORUM (or QUORUM when you must survive a datacenter split with cross-DC replicas), and ALL is reserved for the rare write whose correctness is absolute and whose availability you are willing to sacrifice. Note that writes almost always stay at LOCAL_QUORUM regardless of read band, so W is fixed and you tune R to clear the inequality.

Verify: for each recency-critical class, R + W strictly exceeds N; for each tolerant class, you consciously accepted a sum that may not.

bash 
# Confirm what the driver actually negotiated, not what you think you set
grep -E "read-consistency-level|write-consistency-level" /etc/janusgraph/janusgraph.properties

Step 3 — Handle read-after-write on the index vs the storage path

Storage consistency only governs reads that hit storage. A traversal answered by the mixed index bypasses the CQL level entirely, so a recency-critical read that routes through Elasticsearch or OpenSearch is subject to the index replication window no matter how high you set R. Split the two cases explicitly.

python 
from gremlin_python.driver.driver_remote_connection import DriverRemoteConnection
from gremlin_python.process.anonymous_traversal import traversal
from gremlin_python.process.graph_traversal import __

conn = DriverRemoteConnection("ws://gremlin-01:8182/gremlin", "g")
g = traversal().withRemote(conn)

def read_after_write(entity_id, use_index=False):
    if not use_index:
        # STORAGE PATH: hasId / composite-key lookup obeys storage.cql.read CL.
        # With R + W > N this returns the latest committed write, no index wait.
        return g.V().hasId(entity_id).elementMap().toList()
    # INDEX PATH: a has() on a mixed-indexed property is subject to the index
    # window; storage CL cannot fix it. For these, scope bulk-refresh=wait_for.
    return g.V().has("account", "email", entity_id).elementMap().toList()

The rule is: satisfy read-after-write on the storage path with R + W > N and a hasId() or composite-index lookup, and satisfy it on the index path with bulk-refresh=wait_for scoped to that write — never by raising storage consistency, which does nothing for index visibility. The connection each of these paths holds while it waits is a real resource; a QUORUM read that must reach a slow replica pins its pooled connection longer, which is why consistency choices feed directly into Connection Pooling sizing.

Verify: a just-written record is visible on the storage path immediately and, if it must be visible on the index path, that path uses wait_for.

python 
g.addV("account").property("email", "probe@example.test").iterate()
assert g.V().has("account", "email", "probe@example.test").hasNext() or True  # index may lag
print(g.V().has("account", "email", "probe@example.test").count().next())

Step 4 — Validate the levels under a replica loss

A consistency choice is only proven when you confirm it still returns correct answers with a replica down — that is the exact condition the level exists to survive. Do this on staging, not production.

bash 
# On staging: take one replica offline, then confirm recency-critical reads still work
nodetool -h staging-node-3 disablebinary && nodetool -h staging-node-3 disablegossip

With RF=3 and LOCAL_QUORUM, a single node loss leaves two replicas — the quorum still forms, so recency-critical reads and writes succeed. Confirm your tolerant LOCAL_ONE path also survives (it trivially does), then bring the node back and confirm no stale reads linger past read-repair.

Verify: recency-critical reads succeed with one replica down, and a written value reads back correctly.

bash 
cqlsh staging-node-1 -e "CONSISTENCY LOCAL_QUORUM; SELECT key FROM janusgraph_prod.edgestore LIMIT 1;"
nodetool -h staging-node-3 enablebinary && nodetool -h staging-node-3 enablegossip

Fallback and rollback procedures

Consistency changes are reversible config, but a wrong one corrupts decisions silently, so validate each class before moving on.

If Step 2 leaves a recency-critical class under R + W ≤ N. The read can miss the latest write. Raise R to LOCAL_QUORUM (or QUORUM for cross-DC) so the inequality clears, restart, and re-run the Step 4 validation. Never leave a high-cost class on a sum that does not overlap to save latency — that is trading correctness for milliseconds.

If QUORUM reads start timing out under load. The pool is holding connections longer than it is sized for, because each QUORUM read waits on a slower replica set. Do not drop to LOCAL_ONE reflexively — first confirm whether the class is genuinely tolerant. If it is recency-critical, resize the pool per the Connection Pooling model rather than weakening consistency.

If ALL writes start failing during a rolling restart. ALL requires every replica up, so a single node down fails the write. Roll it back to LOCAL_QUORUM unless the class truly demands absolute correctness; almost none do:

properties 
storage.cql.write-consistency-level=LOCAL_QUORUM

If a datacenter partition makes QUORUM unavailable. A cross-DC QUORUM needs replicas from multiple datacenters and cannot form when they are cut off. Fall back to LOCAL_QUORUM, which counts only local-DC replicas and keeps the local region serving; reconcile cross-DC once the partition heals. The datacenter replica math behind this is in the Replication Strategies reference.

Frequently Asked Questions

What does the R + W > N rule actually guarantee? It guarantees the set of replicas a read consults and the set a write acknowledged overlap in at least one node, so the read is certain to see the most recent committed write. With replication factor N = 3, a QUORUM write (2 replicas) plus a QUORUM read (2 replicas) sums to 4, which exceeds 3, so overlap is guaranteed. It says nothing about mixed-index visibility, which is a separate downstream concern.

Does a high storage consistency level fix stale results from the search index? No. Storage consistency governs durability and read repair inside Cassandra or ScyllaDB only. A traversal answered by the mixed index never consults the CQL consistency level, so raising QUORUM to ALL does not shrink the index replication window. Fix index freshness with bulk-refresh=wait_for on the specific write, or route the read through a storage-backed lookup.

When should I use LOCAL_QUORUM instead of QUORUM? Use LOCAL_QUORUM when your cluster spans datacenters and you want quorum within the local region without paying cross-datacenter round-trip latency. It counts replicas in the local datacenter only, so it stays available during a cross-DC partition. Use plain QUORUM when a read must reflect writes that could have been acknowledged in another datacenter.

Is it safe to run tolerant reads at LOCAL_ONE while writing at LOCAL_QUORUM? Yes, for genuinely tolerant classes. LOCAL_ONE reads the nearest replica for lowest latency and may occasionally return a value that has not yet replicated, which is acceptable for analytical scans, dashboards, and cache warming. Keep the write at LOCAL_QUORUM so the data is durable across replicas even when individual reads sample a single one.