How do I derive the JanusGraph stale-read window as a single metric?

Compose it from three components: sum the IndexProvider queue depth and the Elasticsearch or OpenSearch write queue for total backlog, divide by a clamped rate of index writes to get drain-time in seconds, then add the index refresh interval as fixed dead time. The result, backlog over drain rate plus refresh interval, is the number of seconds a committed vertex can remain invisible to a search.

Why does my index-lag panel never read zero?

Because the refresh interval is a floor. Even with an empty backlog, Elasticsearch and OpenSearch only make a document searchable at the next refresh, so the stale-read window can never fall below the refresh interval. A flat floor equal to the refresh interval is correct behavior, not a bug; confirm it by temporarily removing the refresh term and checking the remainder moves with load.

How do I tell a reindex-caused lag spike from real index drift in Grafana?

Overlay reindex annotations on the same panel. Add a Prometheus annotation query on the reindex job-state signal, or push annotations to the Grafana API at job start and end. A lag spike sitting on top of a reindex marker is expected backlog draining and needs no page; a spike with no marker beneath it is unplanned drift and a real consistency incident.

Why did an index disappear from my summed lag query?

Prometheus vector addition requires matching label sets, so an index present in the IndexProvider queue metric but absent from the Elasticsearch write-queue metric is dropped from the sum. Reconcile the index label naming between the JanusGraph and index-cluster exporters, or combine the terms with on(index) group_left so a one-sided backlog still appears.

Visualizing Index-Sync Lag in Grafana

Index-sync lag is the one JanusGraph signal your users feel directly and your default dashboards almost never show: the interval during which a committed vertex is durable in storage but still invisible to a has()-predicate search. This guide is the procedure for deriving that stale-read window from its three underlying components and drawing it as a single Grafana panel with a threshold line and reindex overlays. It sits under the Grafana Dashboards for JanusGraph reference and builds the Index Lag row that page reserves. The failure it prevents is the silent search-consistency incident — the case where storage panels are green, no error fires, and yet traversals return phantom or missing results because the index is minutes behind and nothing on the wall is measuring it. Every step below is a composable signal you can verify against live Prometheus, with a rollback that leaves the dashboard exactly as it was.

The stale-read window is composed by summing the two queues, dividing by drain rate, and adding the refresh interval, then drawn as a time series with a threshold line and reindex annotation markers.

Prerequisites

Confirm each item before building the panel. Skipping the drain-rate check is the usual reason a derived-lag panel reads as a flat, meaningless number.

A working JanusGraph storage dashboard you can add a panel to. If you have not provisioned one, follow Building a JanusGraph Storage-Backend Dashboard first — this panel joins that dashboard’s Index Lag row.
Prometheus scraping both JanusGraph and the index cluster. You need JanusGraph’s index_provider_queue_depth and index_provider_writes_total plus the Elasticsearch/OpenSearch write thread-pool queue metric (elasticsearch_thread_pool_queue{thread_pool="write"} from the standard ES exporter, or the OpenSearch equivalent).
The index refresh interval as a number. Query the live setting with curl -s http://es:9200/janusgraph_*/_settings | jq '..|.refresh_interval? // empty' and note the value in seconds (the ES/OS default is 1s, but bulk-load profiles often raise it to 30s or disable it).
Reindex jobs that emit a signal. To overlay annotations you need a janusgraph_reindex_job_state gauge or a process that pushes Grafana annotations. If reindexing is manual, you can still add the threshold line and backfill annotations by hand.
Familiarity with your routing chain. Which backing index answers a given query is governed by Mixed-Index Routing; lag on an index that no live query reads is lower priority than lag on the hot path.

Step 1 — Graph the raw backlog

Start with the total backlog, the numerator of the lag formula. Sum the JanusGraph IndexProvider queue with the index cluster’s bulk write queue so a backlog on either side is visible:

promql

sum by (index) (index_provider_queue_depth)
+
sum by (index) (elasticsearch_thread_pool_queue{thread_pool="write"})

This is documents-in-flight, not time. It is a useful early-warning shape but not yet the stale-read window — a backlog of 50 000 is trivial at a high drain rate and catastrophic at a low one. Add it as a temporary panel so you can confirm both terms contribute.

Verify: confirm both component metrics return series and the sum is non-negative.

bash

curl -s http://prometheus:9090/api/v1/query --data-urlencode \
  'query=sum by (index) (index_provider_queue_depth) + sum by (index) (elasticsearch_thread_pool_queue{thread_pool="write"})' \
  | jq '.data.result[] | {index: .metric.index, backlog: .value[1]}'

If one term is missing, the whole sum drops that index (Prometheus + requires matching label sets) — reconcile the index label or use on()/ignoring() before continuing.

Step 2 — Divide by drain rate to get drain-time

Convert the backlog into seconds by dividing by how fast the index is actually applying writes. The drain rate is the per-second rate of successful index writes; clamp it so a zero-rate quiet window does not produce a divide-by-zero:

promql

(
  sum by (index) (index_provider_queue_depth)
  +
  sum by (index) (elasticsearch_thread_pool_queue{thread_pool="write"})
)
/
clamp_min(sum by (index) (rate(index_provider_writes_total[5m])), 1)

Now the panel reads in seconds: “at the current drain rate, the tail of the backlog will be applied this many seconds from now.” This is the part of the stale-read window that responds to load — it grows the instant writes arrive faster than the index drains them.

Verify: confirm the drain-time is a plausible number of seconds, not a spike into the thousands (which signals a near-zero drain rate you should investigate directly).

bash

curl -s http://prometheus:9090/api/v1/query --data-urlencode \
  'query=(sum by (index) (index_provider_queue_depth) + sum by (index) (elasticsearch_thread_pool_queue{thread_pool="write"})) / clamp_min(sum by (index) (rate(index_provider_writes_total[5m])), 1)' \
  | jq '.data.result[].value[1]'

Step 3 — Add the refresh interval and finalize the signal

The drain-time is not the whole story. Even after a document is applied, Elasticsearch and OpenSearch only make it searchable at the next refresh, so the refresh interval $t_r$ is dead time no drain rate can shorten. The complete stale-read window is:

W_{\text{stale}} = \frac{q_{1} + q_{2}}{R} + t_{r}

Add $t_r$ as a scalar to the drain-time. Encode it as a Prometheus recording rule if the interval varies per index, or as a literal in the target if it is uniform:

promql

(
  sum by (index) (index_provider_queue_depth)
  +
  sum by (index) (elasticsearch_thread_pool_queue{thread_pool="write"})
)
/
clamp_min(sum by (index) (rate(index_provider_writes_total[5m])), 1)
+
scalar(elasticsearch_index_refresh_interval_seconds)

Put this in a timeseries panel titled “Index stale-read window (s)” with unit: s. This is the number that answers “how stale can a search be right now,” and it belongs at the top of the Index Lag row. Because the refresh interval dominates when the backlog is small, this panel never reads zero — a floor equal to $t_r$ is correct, not a bug. For how these queues actually form and drain on the OpenSearch side, see OpenSearch Sync Patterns.

Verify: confirm the finalized window is at least the refresh interval and moves with load.

bash

curl -s http://prometheus:9090/api/v1/query --data-urlencode \
  'query=(sum by (index) (index_provider_queue_depth) + sum by (index) (elasticsearch_thread_pool_queue{thread_pool="write"})) / clamp_min(sum by (index) (rate(index_provider_writes_total[5m])), 1) + scalar(elasticsearch_index_refresh_interval_seconds)' \
  | jq '.data.result[].value[1]'

Step 4 — Add the threshold line

A number without a line is just a squiggle. Add an explicit threshold so “acceptable staleness” is visible on the panel without reading the axis. In the panel’s field config, set a step at your consistency SLO — here, five seconds:

json

{
  "fieldConfig": {
    "defaults": {
      "unit": "s",
      "custom": {
        "thresholdsStyle": { "mode": "line+area" }
      },
      "thresholds": {
        "mode": "absolute",
        "steps": [
          { "color": "green", "value": null },
          { "color": "red", "value": 5 }
        ]
      }
    }
  }
}

thresholdsStyle.mode: "line+area" draws a horizontal line at 5 s and shades the breach region, so a glance shows whether the current window is inside SLO. Choose the value from your read-consistency requirement, not from the current baseline — the line is a target, not a description of today.

Verify: load the dashboard and confirm the threshold line renders at 5 s and the series crosses it under synthetic load. Generate load by lowering the drain rate (throttle the index) or replaying a write burst, and watch the area shade red.

Step 5 — Overlay reindex annotations

A lag spike is ambiguous until you know whether a reindex caused it. Add an annotation query so every reindex draws a vertical marker on the same axis:

json

{
  "annotations": {
    "list": [
      {
        "name": "reindex events",
        "enable": true,
        "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" },
        "expr": "changes(janusgraph_reindex_job_state{state=\"RUNNING\"}[2m]) > 0",
        "titleFormat": "reindex {{index}}",
        "textFormat": "job {{job_id}}",
        "tagKeys": "index,job_id"
      }
    ]
  }
}

If your reindex process cannot emit a Prometheus gauge, push annotations directly to Grafana’s API at job start and end:

bash

curl -s -X POST http://localhost:3000/api/annotations \
  -u admin:$GRAFANA_PW -H 'Content-Type: application/json' \
  -d '{"dashboardUID":"janusgraph-storage","tags":["reindex"],
       "text":"reindex started: composite→mixed","time":'"$(date +%s000)"'}'

Now the panel is self-explaining: a window spike sitting on top of a reindex marker is planned backlog draining and needs no page; a spike with no marker beneath it is unplanned drift and a real incident. This distinction is exactly what stops reindex noise from masking true consistency failures.

Verify: trigger a small reindex and confirm a vertical marker appears at its start on the stale-read window panel, aligned with the expected backlog rise.

Fallback and rollback procedures

Validate between actions; each step adds one composable piece you can strip back independently.

If Step 1 drops an index from the sum. Prometheus arithmetic requires matching label sets, so an index present in one term but not the other vanishes from the result. Use sum by (index)(a) + on(index) group_left sum by (index)(b) or reconcile the index label naming between the JanusGraph and ES/OS exporters before layering more on top.

If Step 2 shows a runaway drain-time. A near-zero rate(index_provider_writes_total[5m]) divides a real backlog into a huge number. That is not a graphing bug — it means the index is barely draining. Investigate the index cluster directly (thread-pool rejections, disk saturation) rather than widening the rate window to hide it; the routing implications are covered in Mixed-Index Routing.

If Step 3 reads a constant flat line. The refresh-interval term is dominating because the backlog is genuinely near zero — that is correct behavior. Confirm by temporarily removing the + scalar(...) term; if the remainder moves with load, the panel is fine and the floor is just $t_r$ .

If Step 4’s threshold does not render. Older Grafana panel types ignore thresholdsStyle. Confirm the panel type is timeseries, not graph, and that custom.thresholdsStyle.mode is set — the legacy graph panel needs a separate threshold definition.

If Step 5 annotations do not appear. The annotation query returned nothing because no reindex ran in the window, or the changes() range is shorter than the scrape interval. Widen the changes(...[2m]) range above one scrape interval, and confirm janusgraph_reindex_job_state actually toggles by querying it directly.

If the whole panel must be reverted. Because it is one panel object in a provisioned JSON file, remove it from the panels array and its annotation from the annotations.list, restore from version control, and let the provisioning provider reload. Nothing else on the dashboard is affected. To keep the rest of the board coherent after removing it, return to Building a JanusGraph Storage-Backend Dashboard.

Up a level: Grafana Dashboards for JanusGraph — the dashboard whose Index Lag row this panel builds.
Building a JanusGraph Storage-Backend Dashboard — the sibling procedure that stands up the dashboard this panel joins.
Mixed-Index Routing — which backing index answers a query, and why lag on the hot path matters most.
OpenSearch Sync Patterns — how the bulk queues in this signal actually form and drain.