Grafana Dashboards for JanusGraph

A Grafana dashboard is where a JanusGraph incident is either caught early or missed entirely, and the difference is almost never the graphing tool — it is whether each panel is wired to a signal that actually moves before users do. Under the JanusGraph Monitoring & Observability reference, this page narrows the observability surface to one artifact: the provisioned dashboard that a storage-backend operator keeps open during ingestion, reindexing, and consistency incidents. The failure mode it prevents is the false-green board — a wall of panels that stay flat while the external index drifts, the CQL pool saturates, and stale reads leak into production, because every graph was pointed at a metric that lags the fault instead of leading it. Everything below treats a dashboard as code: rows that map to failure domains, PromQL that expresses a symptom rather than a raw counter, and annotations that let you read cause and effect on the same time axis.

The diagram below maps each backend signal to the dashboard row that surfaces it, so panel structure follows failure structure rather than metric-catalog order.

Dashboard Structure & Panel Rows

A JanusGraph dashboard should read top-to-bottom as a causal chain, not an alphabetical metric dump. When a stale read is reported, the on-call engineer scrolls one screen and sees whether the fault originates in storage, in the index dispatch, in the pool, in the cache, or in the JVM — in that order, because that is the order in which a slow storage commit propagates into a widening index-sync window. Lay the dashboard out as five collapsed rows so the default view fits one screen and each row expands to its detail panels on demand.

Row-ordering constraints, in the order they matter operationally:

1. Storage first, because it is upstream of everything. A rise in storage_cql_request_latency_seconds P99 delays the commit, which delays the index dispatch, which widens the stale-read window and backs up the pool. Putting storage at the top lets you rule the backend in or out in one glance before chasing a downstream symptom.
2. Index Lag second, because it is the symptom users report. Phantom vertices and missing edge properties surface here first. The lead panel must be the derived stale-read window in seconds, not a raw queue count — a queue of 10 000 is meaningless until you multiply by drain rate and refresh interval.
3. Pool Saturation third, because it converts a backend slowdown into an application outage. When in-flight sockets pin at max-connections-per-host, producers block on acquisition. This row is where a storage slowdown becomes a NoHostAvailableException storm.
4. Cache and JVM last, because they are modifiers, not usually root causes. A collapsing cache hit ratio amplifies storage load; a long GC pause freezes acquisition. They explain why the top three rows moved, so they belong at the bottom where you look after localizing the fault.

Keep every panel’s unit explicit (seconds, ratio, ops/s) and every threshold visible as a colored band, so a glance distinguishes “elevated” from “breaching” without reading the axis. The exact numeric bands belong to Alerting Thresholds; the dashboard renders them, the alerting layer fires on them.

The PromQL Behind Each Core Panel

Every panel is only as good as its query. Raw JMX counters are cumulative and per-instance; a useful panel almost always applies rate(), histogram_quantile(), or a ratio to turn a counter into a symptom. The metric names below assume the JanusGraph JMX beans are scraped through the exporter described in the Prometheus & JMX metrics reference; if your exporter renames beans, adjust the label matchers but keep the transforms.

Storage Latency — P99 commit latency. The histogram bucket must be summed by le before the quantile is taken, or you get a per-scrape-target quantile that hides tail latency on the worst node:

histogram_quantile(
  0.99,
  sum by (le, keyspace) (
    rate(storage_cql_request_latency_seconds_bucket{operation="commit"}[5m])
  )
)

Index Lag — derived stale-read window. This is the signal users feel. Model it as the backlog divided by the drain rate, plus the index engine’s own refresh interval, which is dead time no drain rate can shorten. With queue depth $Q$, drain rate $R$ (documents/sec), and refresh interval $t_r$, the visible stale-read window is:

$$W_{\text{stale}} = \frac{Q}{R} + t_r$$

Expressed in PromQL, with the ES/OS refresh interval passed in as a recording rule or a dashboard constant:

(
  sum by (index) (index_provider_queue_depth)
  /
  clamp_min(sum by (index) (rate(index_provider_writes_total[5m])), 1)
)
+ elasticsearch_index_refresh_interval_seconds

Pool Saturation — utilization ratio. Never graph raw in-flight without the denominator; a value of 800 is fine on a pool of 4096 and critical on a pool of 1024:

sum by (host) (cql_pool_in_flight)
/
sum by (host) (cql_pool_max_requests)

Pair it with a NoHostAvailableException rate panel — rate(cql_client_nohostavailable_total[1m]) — because the ratio tells you approaching saturation and the exception rate tells you breached it.

Cache Health — hit ratio. A ratio of rates, never a ratio of raw counters (raw counters bias toward the whole-uptime average and hide a live collapse):

sum(rate(db_cache_hit_total[5m]))
/
clamp_min(sum(rate(db_cache_hit_total[5m])) + sum(rate(db_cache_miss_total[5m])), 1)

JVM / GC — pause time per second. Graph the fraction of wall-clock spent paused, which is directly comparable across heap sizes:

sum by (instance) (rate(jvm_gc_pause_seconds_sum[5m]))
/
sum by (instance) (rate(jvm_gc_pause_seconds_count[5m]))

The clamp_min(..., 1) guard on every denominator is not cosmetic: without it, a panel divides by zero during quiet windows and renders as NaN, which Grafana draws as a gap that looks identical to a scrape outage. Guarding the denominator is the single most common fix for a panel that “goes blank at 3 a.m.”

Provisioned Dashboard JSON

Treat the dashboard as a versioned artifact, not a hand-clicked layout that lives only in Grafana’s database. Provisioning it from a JSON file under source control means the panel targets, thresholds, and datasource binding are reviewable and reproducible across environments. The snippet below is the storage-latency and pool-saturation panels of a provisioned dashboard, showing the target expression and the threshold steps that drive the colored bands:

{
  "title": "JanusGraph — Storage Backend",
  "uid": "janusgraph-storage",
  "templating": { "list": [] },
  "panels": [
    {
      "title": "Storage commit P99 (s)",
      "type": "timeseries",
      "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
      "targets": [
        {
          "refId": "A",
          "expr": "histogram_quantile(0.99, sum by (le, keyspace) (rate(storage_cql_request_latency_seconds_bucket{operation=\"commit\", keyspace=~\"$keyspace\"}[5m])))",
          "legendFormat": "{{keyspace}}"
        }
      ],
      "fieldConfig": {
        "defaults": {
          "unit": "s",
          "thresholds": {
            "mode": "absolute",
            "steps": [
              { "color": "green", "value": null },
              { "color": "yellow", "value": 0.05 },
              { "color": "red", "value": 0.15 }
            ]
          }
        }
      }
    },
    {
      "title": "Pool utilization (in-flight / max)",
      "type": "timeseries",
      "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
      "targets": [
        {
          "refId": "A",
          "expr": "sum by (host) (cql_pool_in_flight{host=~\"$node\"}) / sum by (host) (cql_pool_max_requests{host=~\"$node\"})",
          "legendFormat": "{{host}}"
        }
      ],
      "fieldConfig": {
        "defaults": {
          "unit": "percentunit",
          "max": 1,
          "thresholds": {
            "mode": "absolute",
            "steps": [
              { "color": "green", "value": null },
              { "color": "yellow", "value": 0.7 },
              { "color": "red", "value": 0.9 }
            ]
          }
        }
      }
    }
  ]
}

The two details that make this survive review: the datasource is referenced by the ${DS_PROMETHEUS} variable rather than a hard-coded UID (so the same JSON imports into staging and production without edits), and the target expressions embed the $keyspace and $node template variables directly, so a single dashboard scopes to any node or keyspace without duplicated panels. The step-by-step of turning this file into a live, provisioned dashboard is walked in Building a JanusGraph Storage-Backend Dashboard.

Variables, Templating & Reindex Annotations

Templating is what turns five panels into a fleet-wide tool. Without it you clone the dashboard per node and per keyspace and the copies drift; with it, one dashboard filters live. Define the variables as label-values queries against Prometheus so they populate from whatever is actually being scraped:

{
  "templating": {
    "list": [
      {
        "name": "node",
        "type": "query",
        "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
        "query": "label_values(cql_pool_open_connections, host)",
        "includeAll": true,
        "multi": true
      },
      {
        "name": "keyspace",
        "type": "query",
        "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
        "query": "label_values(storage_cql_request_latency_seconds_count, keyspace)",
        "includeAll": true,
        "multi": true
      }
    ]
  }
}

Two rules keep templating honest. First, use label_values() against a metric that is always present (an _count or an always-emitted gauge), never against one that only appears under load — a variable sourced from cql_client_nohostavailable_total is empty until the first failure, so the dashboard shows no nodes exactly when you need it. Second, set includeAll with a regex-friendly .* all-value and reference the variable as =~"$node" in every target, so the “All” selection is a real query, not a broken literal match.

Annotations wired to reindex events are what let you read cause and effect on one axis. A widening index-lag panel is ambiguous on its own — is the index falling behind, or is a planned reindex deliberately replaying millions of documents? An annotation query resolves it by drawing a vertical marker at each reindex start and end:

{
  "annotations": {
    "list": [
      {
        "name": "reindex events",
        "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
        "expr": "changes(janusgraph_reindex_job_state{state=\"RUNNING\"}[1m]) > 0",
        "titleFormat": "reindex {{index}}",
        "tagKeys": "index,job_id"
      }
    ]
  }
}

Emit a janusgraph_reindex_job_state gauge (or push a Grafana annotation via its HTTP API) from whatever orchestrates your reindex. Once the marker lands on the same time axis as the Index Lag row, a lag spike that starts precisely at a reindex marker is expected backlog draining; a lag spike with no marker beneath it is an unplanned drift and a real incident. The mechanics of composing and reading that lag signal are covered in depth in Visualizing Index-Sync Lag in Grafana.

Python Integration Pattern

A provisioned dashboard is only trustworthy if every panel’s target actually returns data against the live Prometheus instance — a mistyped label matcher renders an empty panel that looks calm rather than broken. The helper below loads the dashboard JSON, extracts every panel target, and executes it against the Prometheus HTTP API, failing loudly on any target that returns no series. Run it in CI whenever the dashboard file changes, so a renamed metric is caught at review time instead of during an incident.

import json
import logging
import requests

logger = logging.getLogger(__name__)


class DashboardTargetValidator:
    def __init__(self, prometheus_url: str, timeout: float = 5.0):
        # prometheus_url points at the same instance Grafana scrapes, so a
        # target that is empty here is empty on the live dashboard too.
        self.base = prometheus_url.rstrip("/")
        self.timeout = timeout

    def _query(self, expr: str) -> list:
        try:
            resp = requests.get(
                f"{self.base}/api/v1/query",
                params={"query": expr},
                timeout=self.timeout,
            )
            resp.raise_for_status()
            payload = resp.json()
        except (requests.RequestException, ValueError) as exc:
            # A transport error is not an empty panel; surface it distinctly so
            # a Prometheus outage is never mistaken for a bad target.
            raise RuntimeError(f"prometheus query failed: {exc}") from exc
        return payload.get("data", {}).get("result", [])

    def _expand(self, expr: str) -> str:
        # Template variables cannot resolve outside Grafana; substitute a
        # permissive matcher so the target is still executable in CI.
        return expr.replace('=~"$node"', '=~".*"').replace('=~"$keyspace"', '=~".*"')

    def validate(self, dashboard_path: str) -> list:
        with open(dashboard_path, encoding="utf-8") as fh:
            dashboard = json.load(fh)
        empty = []
        for panel in dashboard.get("panels", []):
            for target in panel.get("targets", []):
                expr = self._expand(target.get("expr", ""))
                if not expr:
                    continue
                if not self._query(expr):
                    empty.append((panel.get("title", "?"), expr))
                    logger.error("empty target in panel %r: %s",
                                 panel.get("title"), expr)
        return empty


if __name__ == "__main__":
    validator = DashboardTargetValidator("http://prometheus:9090")
    problems = validator.validate("dashboards/janusgraph-storage.json")
    raise SystemExit(1 if problems else 0)

Implementation requirements:

• Substitute template variables with .* before querying; an unexpanded $node is a literal string Prometheus cannot match, producing a false “empty” verdict on an otherwise-correct target.
• Distinguish a transport failure from an empty result. A 500 from Prometheus means the check could not run, not that the panel is bad — conflating them either hides real breakage or floods CI with false failures.
• Guard denominators in the dashboard itself, not in the validator. The validator confirms a target returns series; it cannot confirm the panel renders sensibly when a rate is zero, which is why the clamp_min guards from the PromQL section still matter.

Diagnostics & Operational Fallbacks

Most dashboard incidents are not “the metric is bad” — they are “the panel is lying, and I trusted it.” Triage empty or misleading panels against these symptom/diagnosis/resolution triplets before you conclude the backend is healthy.

• Symptom: A panel is flat green during an active stale-read incident users are reporting. Diagnosis: the panel graphs a raw cumulative counter, not a rate — the counter is still climbing but the value change is invisible at the current axis scale, or the metric lags the fault (e.g. a commit-latency panel while the fault is in index dispatch). Resolution: wrap the target in rate(...[5m]), move the leading signal (index queue depth) above the lagging one, and confirm you are reading the Index Lag row, not Storage Latency, for a search-visibility complaint.

• Symptom: A panel renders gaps that look like a scrape outage but Prometheus targets are all up. Diagnosis: division by zero during a quiet window — the denominator rate hit zero and the expression evaluated to NaN, which Grafana draws as a gap. Resolution: wrap every denominator in clamp_min(..., 1), and set the panel’s “connect null values” to a bounded threshold so a genuine scrape gap still reads differently from an arithmetic hole.

• Symptom: The templated node/keyspace dropdown is empty exactly during an incident. Diagnosis: the variable’s label_values() query is sourced from a metric that only exists under failure (an exception counter), so it disappears when the system is between failures or when the exporter restarts. Resolution: repoint the variable at an always-present _count series or a heartbeat gauge, and cache the last non-empty value with Grafana’s variable refresh set to “On Dashboard Load” rather than “On Time Range Change.”

• Symptom: P99 latency looks fine on the dashboard but individual users see slow queries. Diagnosis: the histogram quantile is computed per-scrape-target and then averaged in the legend, hiding the one hot node — the aggregation dropped the le grouping or aggregated across nodes before the quantile. Resolution: compute histogram_quantile() over sum by (le, host)(...), then graph per-host so the worst node is a visible outlier instead of an average smear.

• Symptom: Index Lag row spikes hard and stays high, but no alert fires and nothing looks broken. Diagnosis: a planned reindex is replaying documents and the spike is expected backlog draining — but the dashboard has no annotation to prove it, so every reindex reads like an incident and real drift hides in the noise. Resolution: wire reindex-event annotations (above) so planned replays are visually attributed; a lag spike with a marker beneath it is normal, one without a marker is the real event to page on.

For the raw metric definitions and JMX bean names behind every query above, keep the Prometheus & JMX metrics reference open alongside this dashboard, and set the numeric fire-points in Alerting Thresholds.

Related

• Up a level: JanusGraph Monitoring & Observability — the observability reference this dashboard renders.
• Building a JanusGraph Storage-Backend Dashboard — step-by-step provisioning of the datasource, panels, and templating shown above.
• Visualizing Index-Sync Lag in Grafana — composing and reading the stale-read window signal on the Index Lag row.
• Prometheus & JMX Metrics Reference — the bean names and exporter behind every PromQL target here.
• Alerting Thresholds — the numeric fire-points the dashboard’s colored bands visualize.