Storage and Index Latency Metrics Reference

This reference catalogues the storage and index latency and throughput metrics worth alerting on in JanusGraph, pairing each one with the MBean it comes from, the range that counts as healthy, and the PromQL that turns a raw timer into a p99 or a rate. It exists to answer the question that stalls most incident reviews: given a slow traversal, which of the dozen janusgraph_* series actually localizes the fault to storage, to the index, or to the transaction layer. The reference sits under the Prometheus JMX Metrics reference, which covers how these beans are exposed; this page is the downstream catalogue you consult once the series are flowing. Adopt the metric set below as a unit — expose every metric, name it consistently, and attach an alert to each — because a partial set leaves exactly the blind spot an incident finds first.

The single most useful mental model is the write-path latency budget: a mutation’s total observed latency is the sum of the storage commit, the index dispatch, and the index refresh, and each metric below measures one slice of it.

The metric set

Every metric below is a series the Prometheus JMX Metrics exporter emits after the rename ruleset runs. Healthy ranges assume a three-node CQL backend at LOCAL_QUORUM; treat them as starting baselines to tighten against your own SLO, not universal constants.

Two formulas underpin the table. The cache hit ratio over a window is one minus the miss rate divided by the retrieval rate:

$$H = 1 - \frac{\Delta_{\text{misses}}}{\Delta_{\text{retrievals}}}$$

The abort ratio, the fraction of transactions that rolled back rather than committed, is:

$$A = \frac{\Delta_{\text{rollback}}}{\Delta_{\text{commit}} + \Delta_{\text{rollback}}}$$

A cache hit ratio drifting below 0.85 or an abort ratio above 0.01 both widen the storage-commit segment of the budget bar, because a cache miss forces a storage read and an abort forces a full transaction replay.

Step 1 — Expose the metric set

Confirm every metric in the table is actually emitted before you build alerts on it — an alert on a series that never appears is a silent gap. The exporter must keep the storage, index, and cache beans, which the step-by-step exporter guide sets up in full.

# Every metric name from the table must return at least one series
for m in janusgraph_store_operation_latency_seconds \
         janusgraph_index_queue_size \
         janusgraph_index_bulk_flush_seconds \
         janusgraph_db_cache_misses_total \
         janusgraph_tx_rollback_total; do
  n=$(curl -s http://localhost:9090/api/v1/query \
       --data-urlencode "query=$m" | jq '.data.result | length')
  printf "%-48s %s series\n" "$m" "$n"
done

Verify: every line reports a non-zero series count. A zero means the corresponding MBean is not being kept — recheck metrics.merge-stores=false and the includeObjectNames allow-list.

Step 2 — Name and normalize consistently

A metric set is only useful if the same concept has the same name and unit across every node. The two normalization rules that matter most are unit scaling and label consistency: all _seconds metrics must actually hold seconds, and the store and backend labels must be present so aggregation works.

# Latency in seconds, not milliseconds — a value near 1000 means a missing valueFactor
janusgraph_store_operation_latency_seconds{operation="mutate", quantile="0.99"}

# Confirm every store carries a label rather than collapsing to one aggregate
count by (store) (janusgraph_store_operation_latency_seconds)

Verify: the count by (store) query returns one row per real store (edgestore, graphindex, janusgraph_ids). A single unlabeled row means merge-stores is still on and the per-store view was lost upstream.

Step 3 — Attach an alert to each metric

Bind a threshold to every row so the set becomes actionable rather than decorative. Use the ratios from the formulas above for the cache and transaction alerts, and a duration guard so a single scrape spike does not page. These fire into the same on-call path described in Alert Routing for Violations.

# prometheus rules
groups:
  - name: janusgraph-latency
    rules:
      - alert: StorageWriteLatencyHigh
        expr: max by (store) (janusgraph_store_operation_latency_seconds{operation="mutate", quantile="0.99"}) > 0.025
        for: 10m
        labels: {severity: warning, tier: storage-backend}
        annotations:
          summary: "p99 storage write latency above 25ms on {{ $labels.store }}"

      - alert: IndexFlushBacklog
        expr: janusgraph_index_queue_size{backend="search"} > 5000
        for: 5m
        labels: {severity: warning}
        annotations:
          summary: "Mixed-index flush queue backing up — sync lag rising"

      - alert: CacheHitRatioLow
        expr: 1 - rate(janusgraph_db_cache_misses_total[5m]) / rate(janusgraph_db_cache_retrievals_total[5m]) < 0.85
        for: 15m
        labels: {severity: info}
        annotations:
          summary: "Database cache hit ratio below 0.85 — storage reads rising"

Verify: the rules load and evaluate without error.

promtool check rules janusgraph-latency.rules.yml
curl -s http://localhost:9090/api/v1/rules | jq '.data.groups[] | select(.name=="janusgraph-latency") | .rules[].name'

Fallback and rollback procedures

Each step has a defined recovery path; validate between actions rather than stacking changes.

If Step 1 shows a zero series count. The metric is not being exported. Confirm the bean exists in jconsole, then check the includeObjectNames list actually names it and metrics.merge-stores=false is set. Do not build an alert on a metric that is not flowing — the alert will read as permanently healthy.

If Step 2 shows collapsed labels or wrong units. A count by (store) returning one row means merging is still on; fix merge-stores and restart the Gremlin Server. Latency values near 1000 mean the valueFactor is missing from the rename rule; add 0.001 for millisecond timers and re-scrape rather than compensating in every PromQL expression.

If Step 3 alerts flap. An alert that fires and clears repeatedly has too tight a threshold or too short a for. Lengthen the for window so a single slow scrape does not page, and widen the threshold toward your true SLO. For lag-based alerts, prefer a rate-of-change guard over a bare level so a slow steady queue does not alert while a genuinely growing one does.

To roll the metric set back. Remove the janusgraph-latency rule group and reload Prometheus; the series remain in the TSDB for historical queries while the alerts stop firing. Reverting the exporter ruleset is covered in the exporter guide’s rollback section.

Related

• Up a level: Prometheus JMX Metrics — how the MBeans behind these series are exposed and renamed.
• Exporting JanusGraph JMX Metrics to Prometheus — the attach-and-verify procedure that makes these series appear.
• OpenSearch Sync Patterns — the index-drift behaviour the queue and flush metrics measure.
• Alert Routing for Violations — where the threshold breaches in Step 3 route on-call.