Storage Backend Selection and Sizing

Picking the wrong storage backend — or the right one sized for a demo instead of production — is the mistake that surfaces months later as an unplanned re-platforming under load, and it is the one decision JanusGraph cannot abstract away for you. The storage adapter you name in storage.backend fixes the write path, the tail-latency floor, the operational burden, and the multi-datacenter story for the life of the graph; changing it after data lands means a full export-and-reload, not a config flip. This reference sits under the JanusGraph Storage Backend Architecture & Configuration reference and narrows it to two coupled questions asked before the first vertex is written: which backend fits the workload, and how many nodes of what shape it takes to hold the graph with room to compact. The failure mode this page prevents is capacity collapse — a storage cluster provisioned from raw dataset size, ignoring the replication multiplier and compaction headroom, that runs out of disk mid-compaction and wedges every node at once. Everything below turns backend choice and node count into arithmetic you can defend in a capacity review.

The decision tree below walks the workload questions that actually change the answer, in the order they eliminate options.

Supported Backends and When Each Fits

JanusGraph speaks to five production-relevant storage adapters, and each occupies a distinct point in the throughput-versus-operability plane.

• Apache Cassandra (storage.backend=cql) is the reference backend and the safe default. Masterless, linearly scalable, and battle-tested for wide-partition graph rows, it fits teams that want a self-hosted store with the largest operational knowledge base and no cloud lock-in. Its cost is a heavier ops burden — compaction tuning, repair scheduling, and JVM garbage collection all land on you.
• ScyllaDB (storage.backend=cql) is a C++ reimplementation of the Cassandra protocol with a shard-per-core reactor architecture. It fits latency-sensitive graphs that must hold a tight P99 under heavy concurrency, because it sidesteps JVM pauses and pins each core to a data shard. It is wire-compatible enough that the CQL adapter reuses the same properties, with the multiplexing values retuned; the ScyllaDB Migration guide covers the driver overrides.
• Apache HBase (storage.backend=hbase) fits organizations already running a Hadoop/HDFS estate who want the graph to share existing region servers and operational tooling. It trades Cassandra’s masterless model for a region-server/HMaster topology with different failure characteristics and a stronger single-row consistency guarantee.
• Google Cloud Bigtable (storage.backend=hbase) reuses JanusGraph’s HBase adapter through the Bigtable HBase client, giving a fully managed, autoscaling store with near-zero node operations. It fits GCP-resident workloads that want to delete the compaction and repair burden entirely and pay for throughput as a managed dimension instead.
• BerkeleyJE (storage.backend=berkeleyje) is an embedded single-JVM store for development, tests, CI, and demos. It fits nothing in production — there is no replication and no horizontal scale — but it is the fastest way to iterate on schema and traversals before a distributed backend is stood up.

The matrix below scores the four production candidates on the dimensions that decide most selections. Read it as a starting bias, then confirm with the capacity math and a pilot.

Core Configuration & Consistency Tuning

Selection and sizing both express themselves through a handful of janusgraph.properties keys read at graph initialization. The block below is the CQL profile you tune from; the replication factor is the single value that multiplies your entire storage estimate, so set it deliberately, not by copying a default.

# Backend selection — the most consequential line in the file
storage.backend=cql                       # cql = Cassandra/ScyllaDB; hbase = HBase/Bigtable
storage.hostname=10.0.1.10,10.0.1.11,10.0.1.12

# Keyspace + replication (CQL backends create the keyspace on first boot)
storage.cql.keyspace=janusgraph_prod
storage.cql.replication-factor=3
storage.cql.replication-strategy-class=org.apache.cassandra.locator.NetworkTopologyStrategy
storage.cql.local-datacenter=us-east-1

# Consistency — quorum bounds the RF latency/durability trade
storage.cql.read-consistency-level=LOCAL_QUORUM
storage.cql.write-consistency-level=LOCAL_QUORUM

# ID allocation block — larger blocks cut coordinator round-trips at load time
ids.block-size=1000000

Constraints that govern how these values interact with capacity, in the order they bite:

1. storage.backend is irreversible in place. cql and hbase write incompatible row layouts. Changing backend after data lands is an export-and-reload, so treat this line as an architecture decision, not a tunable.
2. replication-factor multiplies every byte you store. Raw dataset size is only what one replica holds; the storage cluster stores RF copies. An RF of 3 triples the disk footprint before compaction overhead is even counted. Fix it against your durability and multi-datacenter needs first — the layout choices are covered in Replication Strategies.
3. LOCAL_QUORUM reads and writes both require ⌊RF/2⌋+1 replicas to ack. This makes RF=3 the smallest factor that tolerates one node loss while keeping quorum, which is why 3 is the near-universal starting point for a single datacenter.
4. ids.block-size governs how much of the ID space each JanusGraph instance reserves per round-trip. Under bulk load a small block forces frequent, contended allocations against the janusgraph_ids table; size it to roughly the number of vertices one loader writes between commits.
5. local-datacenter must match nodetool status exactly. A mismatch silently drops the driver to a default topology-unaware policy and skews load, which then corrupts any per-node capacity estimate you made assuming even distribution.

Capacity Sizing Method

Size from the graph, not from a spreadsheet of instance types. Start by estimating the bytes one replica holds, then apply the replication multiplier and compaction headroom, then divide by a target per-node utilization to get node count.

Let V, E, and P be the vertex, edge, and property counts, and b_v, b_e, b_p the average serialized bytes each occupies as a Cassandra row (including the internal column overhead JanusGraph adds — realistic starting estimates are 200 B per vertex row, 400 B per edge row counting both directions, and 60 B per property). The single-replica raw size is:

$$S_{\text{raw}} = V\,b_v + E\,b_e + P\,b_p$$

The storage cluster must hold RF copies, and compaction needs free space to rewrite SSTables — up to the size of the data being compacted under size-tiered compaction. Fold both into the total cluster estimate with a compaction-headroom fraction h_c (about 0.5 for size-tiered, 0.2 for leveled):

$$S_{\text{cluster}} = S_{\text{raw}} \times RF \times (1 + h_c)$$

Then choose node count N so that per-node data stays under a target disk utilization $\rho$ (keep it near 0.6–0.7 so a node loss can be absorbed by streaming without filling a peer):

$$S_{\text{node}} = \frac{S_{\text{cluster}}}{N} \le \rho \, C_{\text{disk}}$$

Solving for the floor on node count, $N \ge S_{\text{cluster}} / (\rho\,C_{\text{disk}})$, rounded up to a multiple that keeps RF racks balanced. Two independent limits then push N higher than the disk floor alone: a per-node data ceiling (Cassandra degrades on repair and streaming past roughly 1–2 TB of data per node; ScyllaDB tolerates more) and a heap constraint — Cassandra wants an 8–16 GB heap with the remaining RAM left to the OS page cache, and a node whose working set no longer fits in page cache pays a disk seek on every read. Size RAM so the hot partition set stays cached, and treat any node that swaps or GC-thrashes as undersized regardless of what the disk math said.

Index Synchronization and the Sizing Interaction

Storage sizing and external-index sizing are separate budgets that fail together. Every mutation JanusGraph commits to storage is dispatched asynchronously to the Elasticsearch or OpenSearch mixed index, so an undersized storage cluster that stalls on compaction backpressure also widens the index sync window — the interval between a durable storage commit and the moment the value is visible through a has() predicate. When storage nodes queue writes behind a compaction backlog, the index dispatch queue drains slower than it fills, and traversals begin returning stale or missing results even though the storage rows are durable.

Size the index tier from the subset of properties that carry a mixed-index mapping, not from P — indexing every property is the most common way to over-provision the index cluster while under-serving storage. Keep the two decoupled during bulk load so index refresh backpressure cannot propagate into the storage commit path, and monitor the dispatch lag as a first-class capacity signal. The acknowledgment-boundary trade-off behind that lag is covered in the index references under External Index Synchronization & Consistency Tuning, and the connection budget that carries every commit is set through Connection Pooling — an undersized pool masquerades as an undersized backend, so reconcile both before adding storage nodes.

Python Integration Pattern

Encode the sizing math as code so a capacity estimate is reproducible and reviewable rather than a stale spreadsheet. The estimator below turns element counts and a backend profile into a raw size, a replicated-plus-compaction cluster size, and a node-count floor, and it fails loudly when inputs are missing rather than silently returning an optimistic number.

from dataclasses import dataclass


@dataclass(frozen=True)
class BackendProfile:
    bytes_per_vertex: int      # serialized row incl. JanusGraph column overhead
    bytes_per_edge: int        # both directions
    bytes_per_property: int
    compaction_headroom: float # 0.5 size-tiered, 0.2 leveled


CASSANDRA_STCS = BackendProfile(200, 400, 60, 0.5)
SCYLLA_ICS = BackendProfile(200, 400, 60, 0.2)


def size_cluster(
    vertices: int,
    edges: int,
    properties: int,
    replication_factor: int,
    profile: BackendProfile,
    disk_per_node_bytes: int,
    target_utilization: float = 0.65,
) -> dict:
    if replication_factor < 1 or not 0 < target_utilization < 1:
        # Fail fast: a bad RF or utilization silently understates node count.
        raise ValueError("replication_factor >= 1 and 0 < utilization < 1 required")

    raw = (
        vertices * profile.bytes_per_vertex
        + edges * profile.bytes_per_edge
        + properties * profile.bytes_per_property
    )
    cluster = raw * replication_factor * (1 + profile.compaction_headroom)
    usable_per_node = target_utilization * disk_per_node_bytes
    node_floor = -(-int(cluster) // int(usable_per_node))  # ceil division

    return {
        "raw_gib": round(raw / 2**30, 1),
        "cluster_gib": round(cluster / 2**30, 1),
        "node_floor": max(node_floor, replication_factor),  # never fewer nodes than RF
    }


if __name__ == "__main__":
    # 500M vertices, 2B edges, 1.5B indexed-ish properties, RF=3, 2 TB SSD/node
    print(
        size_cluster(
            vertices=500_000_000,
            edges=2_000_000_000,
            properties=1_500_000_000,
            replication_factor=3,
            profile=CASSANDRA_STCS,
            disk_per_node_bytes=2 * 2**40,
        )
    )

Implementation requirements:

• bytes_per_* are the estimator’s biggest source of error — measure them against a representative sample by loading 1% of the graph and reading nodetool tablestats rather than trusting the defaults.
• node_floor is a floor, not an answer: raise it further if per-node data would exceed the backend’s practical ceiling or if the working set no longer fits in page cache.
• The max(node_floor, replication_factor) clamp guarantees the storage cluster can place RF distinct replicas — a two-node RF=3 cluster is unsatisfiable and would silently under-replicate.

Diagnostics & Operational Fallbacks

Under-sizing rarely announces itself as “out of capacity”; it presents as latency, GC, or compaction pathology. Triage against these symptom/diagnosis/resolution triplets before adding hardware.

• Symptom: P99 read latency climbs steadily as the dataset grows, with rising disk read IOPS on otherwise idle nodes. Diagnosis: undersized RAM — the working set outgrew the OS page cache, so reads that were served from memory now hit disk. Resolution: move to instances with more RAM (keep the heap at 8–16 GB and leave the rest for page cache), or add nodes to shrink per-node data until the hot partition set fits in cache again.

• Symptom: long stop-the-world GC pauses on storage nodes correlate with request timeouts and connection-pool exhaustion upstream. Diagnosis: undersized or misconfigured heap — too small forces constant collection, too large lengthens each pause and starves the page cache. Resolution: pin the Cassandra heap to 8–16 GB with G1GC, verify the OS has the remaining RAM for cache, and if pauses persist evaluate ScyllaDB, whose C++ runtime removes the JVM pause entirely.

• Symptom: one or two nodes run hot on CPU and disk while peers idle, and a handful of partitions dominate nodetool tablehistograms. Diagnosis: hot partitions — a supernode vertex or a low-cardinality partition key concentrates reads and writes on the replicas owning that token range. Resolution: remodel the schema to shard the supernode (vertex-centric partitioning or edge label sharding), and confirm the partition-size distribution flattens before scaling out, because more nodes do not dilute a single hot partition.

• Symptom: disk usage grows faster than data, pending compactions climb in nodetool compactionstats, and SSTable counts per table run into the hundreds. Diagnosis: compaction backlog — write rate exceeds compaction throughput, so the headroom h_c you budgeted is being consumed by un-merged SSTables. Resolution: raise compaction_throughput_mb_per_sec, switch hot tables to leveled compaction to cut the space amplification, and throttle ingestion until the backlog drains; if it never drains, the storage cluster is write-undersized and needs more nodes.

• Symptom: a single node loss triggers a cascading outage as its replicas fill and refuse writes. Diagnosis: insufficient headroom — nodes ran above the target utilization, so streaming the lost node’s data onto peers exceeded their free space. Resolution: keep per-node utilization near 0.6–0.7 so a node loss can be absorbed, and re-derive node count from S_node ≤ ρ·C_disk with the real replica sizes from nodetool status.

For the storage-tier setup these sizing decisions sit on top of, align through Cassandra Backend Setup, and for the upstream capacity guidance behind the per-node data ceiling see the Apache Cassandra hardware documentation.

Related

• Up a level: JanusGraph Storage Backend Architecture & Configuration — the storage tier this selection and sizing decision provisions.
• Sizing Cassandra Clusters for JanusGraph — the step-by-step capacity procedure with a worked example and per-node validation.
• Choosing Cassandra, ScyllaDB, or Bigtable — a scored decision procedure that maps a workload to a concrete backend and its properties.
• Replication Strategies — replica count and datacenter layout that set the storage multiplier in the sizing math.
• ScyllaDB Migration — the shard-per-core driver overrides for the latency-optimized backend option.
• Connection Pooling — the socket budget that carries every commit, so an undersized pool is not mistaken for an undersized backend.