Sizing Cassandra Clusters for JanusGraph

This procedure computes exactly how many Cassandra nodes, of what instance shape and heap, a JanusGraph deployment needs so the storage cluster holds the graph with room to compact instead of wedging at 90 percent disk mid-merge. It sits under the Storage Backend Selection and Sizing reference and narrows that decision to a single backend: turning measured element counts into a node count you can put in a capacity request and defend with nodetool output. The failure it prevents is the silent under-provision — a storage cluster sized from raw dataset bytes that runs fine in a demo, then hits a compaction storm at scale and cannot free space to complete it, taking every replica down together. Work the steps in order; each one has a verification command, and the byte estimates are corrected against a real sample load rather than trusted blind.

How raw graph bytes expand into a provisioned Cassandra cluster A left-to-right pipeline. Raw single-replica size, computed from vertex, edge, and property byte estimates, is multiplied by the replication factor to give the replicated data, then multiplied by one plus the compaction headroom to give the total on-disk requirement. That total is divided by the usable disk per node, which is target utilization times the raw disk, to yield the node-count floor. The final node count is the larger of that floor and the per-node data ceiling. Raw size V·bᵥ + E·bₑ + P·bₚ one replica Replicated × RF (copies) durability floor × RF On-disk total × (1 + h_c) compaction room × hc Node count ÷ (ρ · disk/node) then ceil to ceiling ÷ node Final node count = max( disk-derived floor , per-node data ceiling ) — the larger constraint wins, and RAM must keep the hot working set in page cache with an 8–16 GB heap reserved for the JVM.
Raw single-replica bytes expand by the replication factor and compaction headroom into an on-disk total, which divides by usable per-node disk to give the node-count floor.

Prerequisites

Confirm each item before you provision anything — a node count derived from guessed byte sizes is worse than no estimate, because it looks authoritative.

  • JanusGraph 0.6.x or 1.0.x targeting a CQL backend with Apache Cassandra 4.x (3.11 works, but 4.x streaming and compaction throughput change the per-node ceiling). If storage is not yet stood up, follow Cassandra Backend Setup first.
  • Element counts for the target graph — vertex, edge, and property totals at the horizon you are sizing for (12–18 months out), not today’s numbers.
  • A representative 1% sample of real data you can load to measure actual bytes per element; synthetic data with uniform property sizes will underestimate wide rows.
  • gremlinpython matching the server’s TinkerPop line, plus nodetool access on every storage node.
  • A decided replication factor. Fix RF and datacenter layout via Replication Strategies before sizing, because it is the multiplier on every byte.

Step 1 — Estimate the dataset size

Estimate the raw size one replica holds. Do not start from disk price or instance catalogs; start from the graph’s element counts and the bytes each element serializes to as a Cassandra row.

python 
# raw single-replica bytes from element counts and measured per-element sizes
def raw_bytes(vertices, edges, properties, b_v=200, b_e=400, b_p=60):
    # b_v/b_e/b_p default to conservative starting estimates in bytes;
    # b_e counts both edge directions JanusGraph writes.
    return vertices * b_v + edges * b_e + properties * b_p


# Worked example: 500M vertices, 2B edges, 1.5B properties
raw = raw_bytes(500_000_000, 2_000_000_000, 1_500_000_000)
print(f"raw single-replica: {raw / 2**40:.2f} TiB")   # ~0.90 TiB

The worked example gives 100 GB + 800 GB + 90 GB = 990 GB ≈ 0.90 TiB for one replica. Treat the default b_v/b_e/b_p as placeholders until Step 4 replaces them with measured values.

Verify: the estimate is dominated by the term you expect (edges here); if a single term is more than ~80% of the total, re-check that count before trusting the size.

bash 
python3 -c "print('edges dominate' if 2_000_000_000*400 > 0.5*990_000_000_000 else 'check inputs')"

Step 2 — Compute per-node storage with RF and compaction headroom

Expand the single-replica size into the on-disk cluster requirement by applying the replication multiplier and compaction headroom, then divide by the usable disk one node offers.

python 
def cluster_and_node_floor(raw, rf=3, h_c=0.5, disk_per_node=2 * 2**40, rho=0.65):
    # rf copies, then compaction headroom (0.5 size-tiered / 0.2 leveled),
    # then divide by usable disk (target utilization rho x raw disk per node).
    cluster = raw * rf * (1 + h_c)
    usable = rho * disk_per_node
    node_floor = -(-int(cluster) // int(usable))     # ceil division
    return cluster, node_floor


cluster, floor = cluster_and_node_floor(990_000_000_000)
print(f"cluster on-disk: {cluster / 2**40:.2f} TiB, node floor: {floor}")
# cluster on-disk: ~4.05 TiB, node floor: 4

With RF=3 and 50% compaction headroom, the 0.90 TiB replica becomes 0.90 × 3 × 1.5 ≈ 4.05 TiB on disk. Against 2 TB SSD nodes at 65% target utilization (≈1.3 TB usable), that is a floor of 4 nodes — but a floor is not the answer until Step 3 checks the per-node data ceiling.

Verify: confirm the projected per-node data sits under the ceiling and the target utilization on a running cluster.

bash 
nodetool status | awk '/^UN/ {print $2, $3}'   # per-node Load; must stay < ~1.3 TB here
nodetool info | grep -E "Load|Heap Memory"

Step 3 — Pick node count, instance type, and heap

Turn the floor into a provisioning spec by reconciling three ceilings: disk (Step 2), per-node data volume, and memory. The largest constraint sets the real node count.

  • Per-node data ceiling. Keep each node under ~1–2 TB of data so repair and streaming stay fast; the 4-node floor already respects this at ~1 TB/node, so disk is the binding constraint here.
  • Instance type. Choose local NVMe SSD over network storage for compaction I/O, at least 8 vCPU, and RAM sized so the hot working set fits in page cache. For the ~1 TB/node example, 64 GB RAM leaves ~48 GB for page cache after a 16 GB heap.
  • Heap. Pin Cassandra to an 8–16 GB heap with G1GC. Bigger is not better — a larger heap lengthens GC pauses and steals RAM from the page cache that actually serves reads.
bash 
# cassandra-env.sh / jvm-server.options — pin, do not autoscale the heap
-Xms16G
-Xmx16G
-XX:+UseG1GC
-XX:MaxGCPauseMillis=300

Round the node count up to keep replica placement balanced across racks — with RF=3, a node count that is a multiple of 3 (or evenly spread across 3 racks) keeps each rack holding one full replica set. Here 4 nodes works with NetworkTopologyStrategy, but many operators pick 6 for clean rack symmetry and headroom.

Verify: confirm the heap took and the page cache is doing the work (low read latency, high cache hit rate).

bash 
nodetool info | grep -E "Heap Memory|Off Heap"
nodetool tablestats janusgraph_prod | grep -E "read latency|bloom filter"

Step 4 — Validate with a load test and correct the estimates

Guessed byte sizes are the biggest error source, so load the 1% sample, measure real bytes per element, and re-run Steps 1–2 with the corrected numbers before committing to the full cluster.

python 
import time
from gremlin_python.driver.driver_remote_connection import DriverRemoteConnection
from gremlin_python.process.anonymous_traversal import traversal


def load_sample(ws_endpoint, vertices=5_000_000, batch=1000):
    conn = DriverRemoteConnection(ws_endpoint, "g")
    g = traversal().withRemote(conn)
    written = 0
    for i in range(vertices // batch):
        gtx = g.tx().begin()
        for n in range(batch):
            gtx.addV("entity").property("k", i * batch + n).property(
                "payload", "x" * 48
            ).iterate()
        gtx.tx().commit()
        written += batch
        if i % 100 == 0:
            time.sleep(0.05)   # let compaction keep pace during the sample load
    conn.close()
    return written


# load_sample("ws://cassandra-canary:8182/gremlin")

After the sample lands and a nodetool flush completes, read the actual on-disk size and back out the true bytes-per-element, then feed them into Step 1.

Verify: compute measured bytes per vertex from live table stats and compare against the b_v=200 assumption.

bash 
nodetool flush janusgraph_prod
nodetool tablestats janusgraph_prod \
  | grep -E "Space used \(total\)|Number of partitions"
# measured_b_v = space_used_total / partitions ; if it exceeds 200, re-size upward
df -h /var/lib/cassandra/data   # confirm disk headroom before scaling the load

If measured bytes exceed the estimate, raise b_v/b_e/b_p and re-derive node count — it is cheaper to add a node on paper now than to reshard a full cluster later.

Fallback and rollback procedures

Validate between actions instead of stacking changes, and keep the sample-load cluster disposable so a wrong estimate costs nothing.

If Step 1 or 2 produces an implausible node count. A count of 1–2 for a large graph usually means an input is off by an order of magnitude — recheck element counts and units (GB vs TiB) before provisioning. Never provision fewer nodes than RF.

If Step 3’s instance runs out of page cache under the sample load. Read latency climbs and disk IOPS spike. Move to a higher-RAM instance or add nodes to shrink per-node data — do not enlarge the heap to “use the RAM”, which lengthens GC pauses and starves the cache further.

If Step 4 shows measured bytes far above the estimate. Stop before scaling the load. Re-run Steps 1–2 with the measured b_* values, re-derive node count, and only then continue; loading the full graph onto an undersized cluster is the failure this procedure exists to prevent.

If a compaction storm fills disk during validation. Throttle ingestion, raise compaction_throughput_mb_per_sec, and if disk keeps climbing, drop the sample keyspace and re-provision with more headroom:

bash 
nodetool disableautocompaction janusgraph_prod   # pause to let manual compaction catch up
nodetool compact janusgraph_prod
# if unrecoverable, tear down the disposable sample cluster and re-size before retrying

If the full rollout must be reverted. Because sizing changes are provisioning, not config, rollback means decommissioning added nodes cleanly with nodetool decommission one at a time so their data streams to peers before removal — never removenode a live node, which skips the stream and under-replicates.