Setting Alert Thresholds for Index Lag

Pinning an index-lag alert to a round number like “30 seconds” guarantees one of two outcomes: it pages on every routine bulk load, or it stays silent while search quietly serves week-old graph state. This guide gives you the arithmetic to pick a threshold that reflects your actual write path — it derives the number from your Elasticsearch or OpenSearch refresh_interval plus your largest bulk-ingest window, writes the PromQL, sets a dwell that ignores load blips, and proves the rule fires with a synthetic lag before you trust it. It sits under Alerting Thresholds and Escalation and narrows that reference to one signal: the write-to-visible latency between a durable graph commit and the moment a has() predicate can see it. The failure this prevents is a stale-read incident that no one is paged for because the threshold was guessed instead of computed.

Deriving an index-lag alert threshold from the refresh interval and bulk window Index synchronization lag rises and falls over time as writes flush to the external index. A steady-state band sits near the refresh_interval. Periodic bulk loads push lag into a higher transient band bounded by refresh_interval plus the bulk-flush window. The alert threshold is drawn above the bulk band with headroom, so routine ingestion never crosses it. When a stuck writer sends lag climbing past the threshold, the breach must persist for the for-duration dwell before the alert transitions from pending to firing, which filters out a single load spike that briefly clips the line. 0 time index lag (s) steady-state ≈ refresh_interval bulk band = refresh + flush window alert threshold bulk load bulk load stuck writer — lag runs away for: dwell → firing

Prerequisites

Confirm each item before writing the rule. Skipping the refresh-interval check is the usual reason a “carefully chosen” threshold still pages on every batch job.

  • JanusGraph 0.6.x or 1.0.x with a mixed index on Elasticsearch 7.x/8.x or OpenSearch 1.x/2.x, and a metric exporting write-to-visible lag as janusgraph_index_write_to_visible_seconds (or derive it — see Step 1). The metric names come from the Prometheus and JMX metrics reference.
  • Prometheus scraping JanusGraph and the index cluster, with promtool on your path for rule validation.
  • A Pushgateway reachable from your workstation for the synthetic test-fire in Step 5.
  • Knowledge of your index refresh_interval. Read it back with GET /<index>/_settings — do not assume the 1s default; production indexes are frequently set to 30s to cut segment churn.
  • Your largest routine bulk-ingest window in seconds — the wall-clock time your biggest scheduled load takes to flush through to the index. If you do not know it, measure one run before setting the threshold.
  • Write access to the Prometheus rule files and permission to reload (SIGHUP or POST /-/reload).

Step 1 — Define the index-lag SLI

The lag you alert on is write-to-visible latency: seconds between a durable storage commit and the point a mixed-index query returns the element. If your build already exports janusgraph_index_write_to_visible_seconds, use it directly. If not, derive it from the index write-queue timestamp versus the last-committed graph mutation timestamp as a recording rule so the alert expression stays cheap:

yaml 
groups:
  - name: janusgraph-index-lag-sli
    rules:
      - record: janusgraph:index_write_to_visible_seconds:max
        expr: |
          max by (index_name) (
            time() - janusgraph_index_last_visible_commit_timestamp_seconds
          )

This is a max, not an average — a single index shard lagging is a stale-read source even if the mean looks healthy. The consistency boundary this SLI measures is exactly the acknowledgment window discussed in Eventual vs Strong Consistency.

Verify: the series exists and is non-negative under normal load.

bash 
curl -s http://localhost:9090/api/v1/query \
  --data-urlencode 'query=janusgraph:index_write_to_visible_seconds:max' \
  | jq '.data.result[] | {index: .metric.index_name, lag: .value[1]}'

Step 2 — Pick a threshold from refresh_interval and the bulk window

The threshold must sit above the highest lag routine operation produces, with headroom, so normal ingestion never pages. Compute it from two measured inputs — the index refresh_interval (steady-state floor) and your largest bulk-flush window (transient ceiling) — and add a headroom multiplier:

threshold=(refresh_interval+bulk_flush_window)×1.5\text{threshold} = (\text{refresh\_interval} + \text{bulk\_flush\_window}) \times 1.5

Worked example: a 30-second refresh_interval and a 20-second bulk-flush window give a bulk band ceiling of 50 seconds; with a 1.5x headroom factor the threshold is 75 seconds. Round to a clean 75. The headroom multiplier absorbs normal variance in flush timing; drop it toward 1.2 only if your bulk window is very stable, and raise it toward 2.0 if batch sizes vary widely.

bash 
# Read the live refresh_interval so the math uses reality, not the default
curl -s 'http://es:9200/janusgraph_vertices/_settings' \
  | jq '.[].settings.index.refresh_interval'

Verify: confirm your chosen threshold sits comfortably above observed peak lag during a real bulk load.

bash 
# During a scheduled bulk load, capture the peak — it must stay under the threshold
curl -s http://localhost:9090/api/v1/query \
  --data-urlencode 'query=max_over_time(janusgraph:index_write_to_visible_seconds:max[15m])' \
  | jq '.data.result[].value[1]'

If peak observed lag during normal ingestion is already near your threshold, the threshold is too tight — raise the headroom factor rather than accepting a rule that will page on the next batch job.

Step 3 — Write the PromQL alert expression

Encode the threshold as an alerting rule. Compare the max SLI against the computed number and attach the routing labels and a runbook up front:

yaml 
groups:
  - name: janusgraph-index-lag
    rules:
      - alert: JanusGraphIndexLagHigh
        expr: janusgraph:index_write_to_visible_seconds:max > 75
        for: 5m
        labels:
          severity: critical
          service: janusgraph
          component: index
        annotations:
          summary: "Index {{ $labels.index_name }} lag {{ $value | humanizeDuration }}"
          description: "Write-to-visible lag exceeded the 75s threshold; search is serving stale graph state."
          runbook: "https://runbooks.internal/janusgraph/index-lag"
      # Missing metric must page too — a stopped exporter reads as healthy otherwise
      - alert: JanusGraphIndexLagMetricAbsent
        expr: absent(janusgraph:index_write_to_visible_seconds:max)
        for: 10m
        labels:
          severity: warning
          service: janusgraph
          component: index
        annotations:
          summary: "Index-lag SLI metric is absent — exporter or scrape is broken"
          runbook: "https://runbooks.internal/janusgraph/metric-absent"

The paired absent() alert is not optional: without it, an exporter that dies makes the lag series vanish and the primary alert reads the missing data as healthy. Which backing index this predicate actually queries — and therefore whose lag you are measuring — depends on the routing described in Mixed-Index Routing.

Verify: the rule parses before you reload Prometheus.

bash 
promtool check rules janusgraph-index-lag.yml

Step 4 — Set the for-duration and severity

The for: clause is what separates a stuck writer from a transient bulk spike. A single bulk load can briefly clip the threshold; requiring the breach to persist filters it out. Use 5m for index lag — long enough to ride out one batch job’s flush, short enough that a genuinely stuck writer pages within the window a stale-read incident becomes user-visible.

Set severity: critical only if stale search results are user-facing. If this index backs an internal analytics path where a few minutes of staleness is tolerable, drop it to warning so it routes to a ticket instead of the pager. The severity label is the routing key the Alertmanager tree in the parent Alerting Thresholds and Escalation reference matches on.

Verify: reload and confirm the rule is registered with the expected for and labels.

bash 
curl -s -XPOST http://localhost:9090/-/reload
curl -s http://localhost:9090/api/v1/rules \
  | jq '.data.groups[].rules[] | select(.name=="JanusGraphIndexLagHigh") | {name, duration, labels}'

Step 5 — Test-fire with a synthetic lag

Never wait for a real incident to learn whether the rule works. Push a synthetic lag value above the threshold through the Pushgateway, watch the alert march from inactive to pending to firing, then clean up.

bash 
# 1. Push a synthetic lag of 90s (above the 75s threshold)
echo 'janusgraph_index_last_visible_commit_timestamp_seconds 0' \
  | curl --data-binary @- \
    http://pushgateway:9091/metrics/job/synthetic_lag/index_name/janusgraph_vertices

# (time() - 0 yields a very large lag, forcing the SLI over threshold)

# 2. Watch the alert state transition over the next few minutes
watch -n 15 "curl -s http://localhost:9090/api/v1/alerts \
  | jq '.data.alerts[] | select(.labels.alertname==\"JanusGraphIndexLagHigh\") | {state, activeAt}'"

After the for: 5m dwell elapses the alert must show "state": "firing". Confirm the page actually reached the pager (or a test receiver), then remove the synthetic series so it does not linger and re-fire:

bash 
curl -s -X DELETE \
  http://pushgateway:9091/metrics/job/synthetic_lag/index_name/janusgraph_vertices

Verify: after deletion, the alert returns to inactive within one evaluation-plus-scrape cycle.

bash 
curl -s http://localhost:9090/api/v1/alerts \
  | jq '.data.alerts[] | select(.labels.alertname=="JanusGraphIndexLagHigh") | .state'

Fallback and rollback procedures

Validate between actions rather than stacking changes.

If Step 2 gives a threshold below observed peak lag. Your bulk window measurement is stale or batch sizes grew. Re-measure the largest routine load with max_over_time(...[30m]), raise the headroom multiplier to 2.0, and recompute — do not ship a rule you already know pages on normal traffic.

If Step 3 fails promtool. A YAML indentation error or an unescaped template usually causes it. Fix the reported line, re-run promtool check rules, and only reload once it passes; a broken rule file makes Prometheus refuse the entire reload, not just the bad rule.

If Step 5 never reaches firing. Either the synthetic metric is not being scraped (confirm the Pushgateway target is up in Prometheus), the recording rule feeding the SLI is not evaluating, or the for: has not elapsed. Query the raw pushed series directly before blaming the alert.

If the new threshold pages spuriously in production. Roll back by restoring the previous rule file from version control and reloading; then widen the headroom factor or lengthen for: to 10m and re-canary against a staging Prometheus before re-applying. Never silence the alert as a fix — a silenced lag alert is indistinguishable from a working one right up until the stale-read incident.