Building a JanusGraph Storage-Backend Dashboard

This guide walks the exact steps to stand up a provisioned Grafana dashboard for a JanusGraph storage backend — from wiring the Prometheus datasource to shipping storage-latency, pool-utilization, and cache-hit panels that carry the right PromQL and threshold colors. It sits under the Grafana Dashboards for JanusGraph reference and turns that page’s row structure into files you commit and deploy. The failure this procedure prevents is the click-built dashboard that exists only in one Grafana instance’s database: it cannot be reviewed, drifts silently between staging and production, and disappears the day someone rebuilds the pod. Every artifact below is a file under version control with a verification you can run before moving on, so the dashboard is reproducible rather than remembered.

How provisioning files wire a Prometheus datasource into templated Grafana panels Two provisioning YAML files on the left, a datasource file and a dashboard-provider file, are read by Grafana at startup. The datasource YAML registers the Prometheus datasource. The dashboard-provider YAML points Grafana at a folder of dashboard JSON files. The dashboard JSON defines three panels — storage commit P99, pool utilization, and cache hit rate — each carrying a PromQL target and threshold steps. Node and keyspace template variables sit above the panels and are substituted into every target so one dashboard scopes to any node or keyspace. All three panels query the single registered Prometheus datasource. PROVISIONING YAML datasources.yaml registers Prometheus dashboards.yaml points at JSON folder Prometheus datasource uid: DS_PROMETHEUS janusgraph-storage.json template variables  $node · $keyspace → substituted into every target Storage P99 histogram_quantile unit: s thr 0.05 / 0.15 Pool util in_flight / max unit: % thr 0.7 / 0.9 Cache hit rate(hit)/rate(all) unit: % thr 0.8 / 0.6 all panels query one datasource
Two provisioning YAML files register the datasource and load the dashboard JSON; template variables substitute into three panel targets that all query the single Prometheus datasource.

Prerequisites

Confirm every item before writing a single provisioning file. The most common cause of a “blank dashboard” is a datasource UID mismatch that no amount of panel editing fixes.

  • Grafana 9.x or 10.x with filesystem access to its provisioning directory (/etc/grafana/provisioning/ by default). Provisioning is disabled for dashboards created in the UI, so plan to manage this dashboard exclusively from files.
  • A Prometheus instance already scraping JanusGraph through the JMX exporter. If metrics are not yet flowing, stand that up first via the Prometheus & JMX metrics reference — this procedure visualizes metrics, it does not produce them.
  • Confirmed metric names. Run curl -s http://prometheus:9090/api/v1/label/__name__/values | jq and verify storage_cql_request_latency_seconds_bucket, cql_pool_in_flight, db_cache_hit_total, and db_cache_miss_total are present. Exporter renames will break every target below.
  • Write access to the provisioning folders and permission to restart or reload Grafana so it re-reads the files.
  • Decision on threshold bands. The colored steps below are starting points; reconcile them against your own SLOs and the fire-points in Alerting Thresholds before promoting to a shared instance.

Step 1 — Provision the Prometheus datasource

Register the datasource from a file so every dashboard can reference it by a stable UID instead of a per-instance database ID. Write /etc/grafana/provisioning/datasources/prometheus.yaml:

yaml 
apiVersion: 1
datasources:
  - name: Prometheus
    uid: DS_PROMETHEUS
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    jsonData:
      httpMethod: POST
      timeInterval: 15s

The uid: DS_PROMETHEUS is the load-bearing line: the dashboard JSON references ${DS_PROMETHEUS}, so pinning the UID here is what lets the same dashboard file import unchanged across environments. Set timeInterval to your Prometheus scrape interval so Grafana’s $__rate_interval math is correct.

Verify: reload Grafana and confirm the datasource registered and is reachable.

bash 
curl -s -u admin:$GRAFANA_PW \
  http://localhost:3000/api/datasources/uid/DS_PROMETHEUS | jq '.name, .type, .url'

A null response means the file was not read — check that provisioning is enabled and the path is correct before continuing.

Step 2 — Provision the dashboard folder

Tell Grafana where dashboard JSON files live. Write /etc/grafana/provisioning/dashboards/janusgraph.yaml:

yaml 
apiVersion: 1
providers:
  - name: janusgraph
    orgId: 1
    folder: JanusGraph
    type: file
    disableDeletion: false
    updateIntervalSeconds: 30
    allowUiUpdates: false
    options:
      path: /var/lib/grafana/dashboards/janusgraph
      foldersFromFilesStructure: false

Set allowUiUpdates: false deliberately — it forces every change to go through the file, which is the entire point of provisioning. Create the target directory and drop a minimal dashboard shell at /var/lib/grafana/dashboards/janusgraph/janusgraph-storage.json:

json 
{
  "uid": "janusgraph-storage",
  "title": "JanusGraph — Storage Backend",
  "schemaVersion": 39,
  "templating": { "list": [] },
  "panels": []
}

Verify: confirm Grafana loaded the empty dashboard by UID.

bash 
curl -s -u admin:$GRAFANA_PW \
  http://localhost:3000/api/dashboards/uid/janusgraph-storage | jq '.dashboard.title'

Step 3 — Add the storage-latency panel

Add the first real panel to the panels array. This graphs P99 commit latency, the leading storage signal. Summing the histogram buckets by le before taking the quantile is mandatory — skip it and you get a per-target quantile that averages away your worst node:

json 
{
  "id": 1,
  "title": "Storage commit P99 (s)",
  "type": "timeseries",
  "gridPos": { "h": 8, "w": 12, "x": 0, "y": 0 },
  "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
  "targets": [
    {
      "refId": "A",
      "expr": "histogram_quantile(0.99, sum by (le, host) (rate(storage_cql_request_latency_seconds_bucket{operation=\"commit\", host=~\"$node\", keyspace=~\"$keyspace\"}[5m])))",
      "legendFormat": "{{host}}"
    }
  ],
  "fieldConfig": {
    "defaults": {
      "unit": "s",
      "thresholds": {
        "mode": "absolute",
        "steps": [
          { "color": "green", "value": null },
          { "color": "yellow", "value": 0.05 },
          { "color": "red", "value": 0.15 }
        ]
      }
    }
  }
}

The threshold steps drive the colored bands: green under 50 ms, yellow from 50 ms, red from 150 ms. Graphing per host keeps the worst node a visible outlier rather than an averaged smear.

Verify: run the raw target against Prometheus and confirm it returns series, not an empty array.

bash 
curl -s http://prometheus:9090/api/v1/query --data-urlencode \
  'query=histogram_quantile(0.99, sum by (le, host) (rate(storage_cql_request_latency_seconds_bucket{operation="commit"}[5m])))' \
  | jq '.data.result | length'

Step 4 — Add the pool-utilization panel

This panel converts the CQL pool from a mystery into a ratio. Always divide by the max; a raw in-flight count is uninterpretable without its ceiling:

json 
{
  "id": 2,
  "title": "Pool utilization (in-flight / max)",
  "type": "timeseries",
  "gridPos": { "h": 8, "w": 12, "x": 12, "y": 0 },
  "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
  "targets": [
    {
      "refId": "A",
      "expr": "sum by (host) (cql_pool_in_flight{host=~\"$node\"}) / clamp_min(sum by (host) (cql_pool_max_requests{host=~\"$node\"}), 1)",
      "legendFormat": "{{host}}"
    }
  ],
  "fieldConfig": {
    "defaults": {
      "unit": "percentunit",
      "max": 1,
      "thresholds": {
        "mode": "absolute",
        "steps": [
          { "color": "green", "value": null },
          { "color": "yellow", "value": 0.7 },
          { "color": "red", "value": 0.9 }
        ]
      }
    }
  }
}

The clamp_min(..., 1) on the denominator is what stops the panel rendering a NaN gap during idle windows. Yellow at 70% utilization is your “widen the pool soon” signal; red at 90% is “acquisition is about to fail.”

Verify: confirm the ratio stays within [0,1] under current load.

bash 
curl -s http://prometheus:9090/api/v1/query --data-urlencode \
  'query=sum by (host) (cql_pool_in_flight) / clamp_min(sum by (host) (cql_pool_max_requests), 1)' \
  | jq '.data.result[].value[1]'

Step 5 — Add the cache-hit panel

A collapsing cache hit ratio amplifies storage load, so it earns a panel. Compute a ratio of rates, never a ratio of raw counters — raw counters bias toward the whole-uptime average and hide a live collapse:

json 
{
  "id": 3,
  "title": "DB cache hit rate",
  "type": "timeseries",
  "gridPos": { "h": 8, "w": 12, "x": 0, "y": 8 },
  "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
  "targets": [
    {
      "refId": "A",
      "expr": "sum(rate(db_cache_hit_total{host=~\"$node\"}[5m])) / clamp_min(sum(rate(db_cache_hit_total{host=~\"$node\"}[5m])) + sum(rate(db_cache_miss_total{host=~\"$node\"}[5m])), 1)",
      "legendFormat": "hit rate"
    }
  ],
  "fieldConfig": {
    "defaults": {
      "unit": "percentunit",
      "max": 1,
      "thresholds": {
        "mode": "absolute",
        "steps": [
          { "color": "red", "value": null },
          { "color": "yellow", "value": 0.6 },
          { "color": "green", "value": 0.8 }
        ]
      }
    }
  }
}

Note the inverted threshold order: for a hit ratio, low is bad, so red is the base color and green begins at 0.8. Getting the direction wrong is a classic cache-panel bug that paints a failing cache green.

Verify: confirm the hit rate is a sensible fraction and not stuck at zero or one.

bash 
curl -s http://prometheus:9090/api/v1/query --data-urlencode \
  'query=sum(rate(db_cache_hit_total[5m])) / clamp_min(sum(rate(db_cache_hit_total[5m])) + sum(rate(db_cache_miss_total[5m])), 1)' \
  | jq '.data.result[].value[1]'

Step 6 — Add templating variables and save

Templating turns three static panels into a fleet tool. Replace the empty templating.list with node and keyspace variables sourced from always-present metrics:

json 
{
  "templating": {
    "list": [
      {
        "name": "node",
        "type": "query",
        "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
        "query": "label_values(cql_pool_open_connections, host)",
        "includeAll": true,
        "allValue": ".*",
        "multi": true,
        "refresh": 1
      },
      {
        "name": "keyspace",
        "type": "query",
        "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
        "query": "label_values(storage_cql_request_latency_seconds_count, keyspace)",
        "includeAll": true,
        "allValue": ".*",
        "multi": true,
        "refresh": 1
      }
    ]
  }
}

refresh: 1 means “on dashboard load,” which keeps the last node list even if the source metric briefly disappears. The allValue: ".*" paired with the =~"$node" matchers in every target is what makes the “All” selection a real regex query. Because allowUiUpdates is false, saving is just writing the file — commit it and let the provisioning provider reload it within updateIntervalSeconds.

Verify: confirm all three panels and both variables landed, and that the variables populate.

bash 
curl -s -u admin:$GRAFANA_PW \
  http://localhost:3000/api/dashboards/uid/janusgraph-storage \
  | jq '.dashboard.panels | length, (.dashboard.templating.list[].name)'

You should see 3 panels and the names node and keyspace. Open the dashboard and confirm the dropdowns list your live nodes and keyspaces.

Fallback and rollback procedures

Validate between actions rather than stacking changes; a broken provisioning file can wedge Grafana’s startup.

If Step 1 fails (datasource not registered). Grafana logs error parsing provisioning file at startup. Validate the YAML with yamllint, confirm apiVersion: 1 is present, and check that no other datasource file already claims DS_PROMETHEUS — a duplicate UID makes Grafana skip the file silently.

If Step 2 fails (dashboard not loaded). A malformed dashboard JSON is rejected whole. Run jq empty janusgraph-storage.json to catch syntax errors, and confirm the path in the provider YAML matches the directory the JSON actually lives in.

If any panel in Steps 3–5 renders empty. The target’s label matchers do not match live series. Strip the {...} matchers down to the bare metric name in Grafana’s query inspector, confirm data appears, then add matchers back one at a time to find the one that misses. A NaN gap rather than an empty panel means an unguarded denominator — add clamp_min(..., 1).

If Step 6 leaves empty dropdowns. The variable’s source metric is absent right now. Repoint label_values() at a metric you confirmed present in Prerequisites, and never source a variable from an exception counter that only exists under failure.

If the whole dashboard must be reverted. Because it is provisioned from a file, rollback is a version-control operation: restore the previous JSON, let the provider reload it, and confirm the UID still resolves. There is no orphaned UI state to clean up — the file is the source of truth. To read and derive the index-lag row that belongs on this dashboard next, continue with Visualizing Index-Sync Lag in Grafana.