Alerting Thresholds & Escalation

An alert that fires at the wrong level is worse than no alert at all: a page on a metric nobody can act on trains on-call to ignore the pager, and a threshold set to a round number teaches the graph to break just below it. This reference sits under JanusGraph Monitoring and Observability and narrows it to one discipline — deciding which storage-backend and index signals cross the line from “watch on a dashboard” into “wake a human,” then encoding that decision as Prometheus alerting rules and Alertmanager routes that survive a real incident. The failure mode this page prevents is alert erosion: the slow slide where index-lag pages, connection-pool pages, and Cassandra-node pages all fire together during one root cause, the on-call engineer silences the lot, and the next genuine outage arrives to a muted queue. Everything below prioritizes a small number of symptom-based, budget-aware, de-duplicated alerts over a wall of threshold triggers on raw counters.

The diagram traces one signal from raw metric to a delivered page: it is evaluated against a threshold model, held for a dwell window, matched to a severity, then routed or suppressed.

The path from a raw JanusGraph signal to a routed page or suppressed alert Five raw signals — index synchronization lag, CQL pool in-flight ratio, cache-hit ratio, transaction-abort rate, and storage node up — feed a threshold model that can be static, rate-of-change, or multi-window burn-rate. A firing candidate must persist through the for-duration dwell gate before it becomes a pending-to-firing alert; the dwell window is what kills flapping. Alertmanager then matches the alert's severity label and either routes a critical alert to the pager, a warning to a ticket queue, or inhibits a downstream alert when a root-cause alert for the same cluster is already firing, preventing one incident from paging five times. SIGNALS index sync lag pool in-flight ratio cache-hit ratio tx-abort rate storage node up Threshold model static bound rate-of-change burn-rate (SLO) expr → pending for: dwell pending → firing kills flapping ROUTE / ESCALATE critical → page pager, 0s group-wait warning → ticket chat, business hours inhibited → suppress root cause already firing crosses severity label decides the route

Which Signals Deserve an Alert, and Which Belong on a Dashboard

The first decision is not what threshold to set — it is whether the signal should ever page. Apply one test: could the person you wake take an action that changes the outcome in the next fifteen minutes? If not, the signal belongs on a Grafana dashboard as context, not on the pager. This is the difference between symptom alerts and cause alerts. Users experience symptoms — traversals timing out, search returning stale results — so symptom alerts have a high action-to-noise ratio. Causes such as a single elevated GC pause or a transient cache dip are diagnostic breadcrumbs; alerting on every cause produces a page storm during one incident.

For a JanusGraph deployment on Cassandra or ScyllaDB with an Elasticsearch or OpenSearch index, sort the raw metrics from the Prometheus and JMX metrics reference into three tiers:

Signal Tier Why
Index synchronization lag (write-to-visible seconds) Page Directly maps to stale search results a user sees
CQL pool in-flight / max saturation Page Predicts imminent NoHostAvailableException and commit stalls
Cassandra/ScyllaDB node up == 0 Page Quorum risk; loss of a second node is an outage
Transaction abort rate spike Page Signals lock contention or a broken write path
Storage read/write P99 latency Dashboard first, page on SLO burn Noisy alone; meaningful as an SLO
Cache-hit ratio collapse Page (rate-of-change) Predicts a latency cliff before P99 moves
Heap usage, GC pause count Dashboard Diagnostic context, rarely actionable in isolation
Compaction pending tasks Dashboard Slow-moving; ticket, not page

The rule that keeps the pager credible: alert on the handful of signals a user would feel, and leave everything else on dashboards where you look during the investigation. A single node being down pages; a single node’s GC pause does not.

Threshold Models: Static, Rate-of-Change, and Burn-Rate

Three threshold shapes cover almost every JanusGraph alert, and choosing the wrong shape is the usual reason an alert is either deaf or hysterical.

  • Static bound — fire when a value crosses a fixed line (up == 0, pool saturation > 0.9). Correct when the safe operating envelope is known and stable. Wrong for anything whose baseline drifts with traffic, because a static line calibrated for the night is either deaf at peak or screaming at 3 a.m.
  • Rate-of-change — fire on the derivative, not the level (deriv(cache_hit_ratio[10m]) < -0.05). Correct for signals where the speed of movement is the warning, like a cache-hit ratio falling off a cliff before latency has caught up.
  • Burn-rate — fire when you are consuming an error budget faster than the SLO allows. Correct for latency and availability, where no single scrape is “bad” but the sustained rate is. This is the model that lets one alert be both fast for hard outages and slow for gentle degradations.

The following janusgraph-alerts.yml encodes the five page-worthy storage-and-index signals. Load it into Prometheus via rule_files: and confirm it parses with promtool check rules janusgraph-alerts.yml before reload.

yaml 
groups:
  - name: janusgraph-storage-index
    rules:
      # 1. Index sync lag — static ceiling derived from refresh_interval + bulk window
      - alert: JanusGraphIndexLagHigh
        expr: janusgraph_index_write_to_visible_seconds > 30
        for: 5m
        labels:
          severity: critical
          service: janusgraph
          component: index
        annotations:
          summary: "Index sync lag {{ $value | humanizeDuration }} on {{ $labels.index_name }}"
          description: "Write-to-visible lag exceeds 30s; search returns stale graph state."
          runbook: "https://runbooks.internal/janusgraph/index-lag"

      # 2. CQL pool saturation — in-flight vs configured max
      - alert: JanusGraphPoolSaturated
        expr: |
          janusgraph_cql_pool_in_flight
            / janusgraph_cql_pool_max_in_flight > 0.9
        for: 3m
        labels:
          severity: critical
          service: janusgraph
          component: storage
        annotations:
          summary: "CQL pool {{ $labels.host }} at {{ $value | humanizePercentage }} in-flight"
          runbook: "https://runbooks.internal/janusgraph/pool-saturation"

      # 3. Cache-hit collapse — rate-of-change, fires before the latency cliff
      - alert: JanusGraphCacheHitCollapse
        expr: |
          janusgraph_db_cache_hit_ratio < 0.6
            and deriv(janusgraph_db_cache_hit_ratio[10m]) < -0.03
        for: 10m
        labels:
          severity: warning
          service: janusgraph
          component: storage
        annotations:
          summary: "DB cache-hit ratio falling fast on {{ $labels.instance }}"
          runbook: "https://runbooks.internal/janusgraph/cache-collapse"

      # 4. Transaction abort spike — rate over a short window vs commit rate
      - alert: JanusGraphTxAbortSpike
        expr: |
          rate(janusgraph_tx_abort_total[5m])
            / clamp_min(rate(janusgraph_tx_commit_total[5m]), 1) > 0.1
        for: 5m
        labels:
          severity: critical
          service: janusgraph
          component: storage
        annotations:
          summary: "{{ $value | humanizePercentage }} of transactions aborting"
          runbook: "https://runbooks.internal/janusgraph/tx-abort"

      # 5. Storage node down — static, short dwell, this is the root-cause alert
      - alert: StorageNodeDown
        expr: up{job="cassandra"} == 0
        for: 1m
        labels:
          severity: critical
          service: janusgraph
          component: storage-node
        annotations:
          summary: "Storage node {{ $labels.instance }} is down"
          runbook: "https://runbooks.internal/cassandra/node-down"

Constraints, in the order they bite:

  1. Every alert carries a runbook annotation. A page without a runbook link is a page the half-asleep responder cannot act on; make the link a hard requirement in review, not an afterthought.
  2. clamp_min on the denominator prevents divide-by-zero flaps. When commit rate briefly hits zero during a deploy, an unclamped ratio evaluates to NaN or +Inf and fires spuriously.
  3. severity is a routing key, not a description. Alertmanager routes on it; treat critical and warning as a contract with the routing tree, not free-text.
  4. The node-down alert is deliberately the shortest dwell. It is the root cause that inhibits the downstream storage alerts, so it must win the race to fire — more on that in the routing section.

Multi-Window Burn-Rate SLO Alerting

Latency and availability alerts are where static thresholds fail hardest, because the same P99 number is fine at 2 a.m. and catastrophic under load. Frame them against an SLO instead. Define the storage-read SLO as a target — say 99.5% of index-backed reads return within 200 ms over 30 days. The error budget is the fraction of requests you are permitted to fail:

error budget=1−SLO target\text{error budget} = 1 - \text{SLO target}

The burn rate is how fast you are spending that budget relative to spending it evenly across the window:

burn rate=observed error ratio1−SLO target\text{burn rate} = \frac{\text{observed error ratio}}{1 - \text{SLO target}}

A burn rate of 1 exhausts the budget exactly at the window’s end; a burn rate of 14.4 exhausts a 30-day budget in roughly two days. Multi-window alerting pairs a fast window (catches hard outages in minutes) with a slow window (catches gentle degradation) and requires both to breach, which is what suppresses single-spike noise:

yaml 
- name: janusgraph-slo-burn
  rules:
    - alert: StorageReadFastBurn
      expr: |
        (
          janusgraph:read_error_ratio:rate5m
            / (1 - 0.995) > 14.4
        )
        and
        (
          janusgraph:read_error_ratio:rate1h
            / (1 - 0.995) > 14.4
        )
      for: 2m
      labels:
        severity: critical
        component: storage-slo
      annotations:
        summary: "Storage-read error budget burning 14.4x — page now"
        runbook: "https://runbooks.internal/janusgraph/slo-fast-burn"

    - alert: StorageReadSlowBurn
      expr: |
        (
          janusgraph:read_error_ratio:rate30m
            / (1 - 0.995) > 3
        )
        and
        (
          janusgraph:read_error_ratio:rate6h
            / (1 - 0.995) > 3
        )
      for: 15m
      labels:
        severity: warning
        component: storage-slo
      annotations:
        summary: "Storage-read budget burning 3x over 6h — investigate before it escalates"
        runbook: "https://runbooks.internal/janusgraph/slo-slow-burn"

The janusgraph:read_error_ratio:rateXm series are recording rules — precompute the ratio at scrape time so the alert expression stays cheap and the burn-rate arithmetic is centralized. The pairing is the whole point: a fast window alone re-fires on every transient spike, and a slow window alone takes hours to notice a hard outage. Requiring both keeps the fast-burn page trustworthy while the slow-burn warning gives you a business-hours ticket before a soft regression becomes a customer-visible one.

Alertmanager Routing, Inhibition, and Escalation

A firing alert is a candidate, not a page. Alertmanager decides who hears it, when, and whether it is suppressed because a more fundamental alert is already active. The routing tree below sends storage-and-index criticals to the pager immediately, warnings to a chat channel during business hours, and — critically — inhibits the downstream storage alerts when a node is already known down.

yaml 
route:
  receiver: default-chat
  group_by: ['service', 'component', 'cluster']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers:
        - service = "janusgraph"
        - severity = "critical"
      receiver: pagerduty-oncall
      group_wait: 0s          # page storage/index criticals with no batching delay
      continue: false
    - matchers:
        - service = "janusgraph"
        - severity = "warning"
      receiver: slack-graph-team
      active_time_intervals: [business-hours]

inhibit_rules:
  # A downed storage node is the root cause; suppress the pool/lag/abort noise it causes
  - source_matchers:
      - alertname = "StorageNodeDown"
    target_matchers:
      - component =~ "storage|index"
      - severity = "critical"
    equal: ['cluster']
  # SLO fast-burn subsumes the slow-burn warning for the same component
  - source_matchers:
      - alertname = "StorageReadFastBurn"
    target_matchers:
      - alertname = "StorageReadSlowBurn"
    equal: ['component']

time_intervals:
  - name: business-hours
    time_intervals:
      - weekdays: ['monday:friday']
        times:
          - start_time: '09:00'
            end_time: '18:00'

Escalation is layered on top of routing at the receiver: PagerDuty (or your pager) escalates to a secondary responder if the primary does not acknowledge within a fixed window, and to the team lead after that. Encode the acknowledgment SLA in the pager policy, not in Prometheus — Prometheus decides what is wrong, the pager decides who answers and by when. The equal: ['cluster'] clause on the inhibition rule is load-bearing: it scopes suppression to the graph deployment whose node is down, so a downed node in cluster A never silences a real pool-saturation page in cluster B. This same source-then-suppress pattern for schema-tier signals is documented in Alert Routing for Violations, and the storage-saturation half of it is worked end-to-end in Paging Rules for Storage-Backend Saturation.

Tuning for: to Kill Flapping

The for: clause is the single most effective flap control you have, and the most commonly left at its accidental value. It requires the expression to stay true continuously for the whole duration before the alert transitions from pending to firing. Too short and every scrape-level spike pages; too long and a real outage burns minutes of undetected damage before the page lands.

Size for: from the signal’s natural jitter and the cost of a late page:

  • Node down: 1m. The cost of a false alarm is low, the cost of a late page is an outage — dwell just long enough to ride out a scrape gap or a rolling restart.
  • Pool saturation: 3m. Bursts self-resolve in seconds; a saturation that persists three minutes is a real capacity problem.
  • Index lag: 5m. A single bulk load spikes lag transiently; five minutes distinguishes a load blip from a stuck writer.
  • Cache-hit collapse: 10m. Slow-moving and paired with a rate condition, so a longer dwell avoids paging on cache warm-up after a restart.

Two anti-flap complements to for:: add hysteresis by making the fire and clear thresholds different (fire at > 0.9, and because for: already delays firing, the resolve on drop-below is naturally damped), and never alert on a raw gauge that oscillates around the threshold — smooth it with avg_over_time(...[5m]) first. If an alert still flaps after a sane for:, the expression is measuring jitter, not a state.

Python Integration Pattern

Before an alert ships, prove it fires on the condition it claims to catch and stays quiet otherwise. This harness queries the Prometheus HTTP API to replay a rule’s expr over a historical window, then test-fires it against a synthetic series so you can validate a new threshold without waiting for a real incident. Pair every new alert with a run of this before merge.

python 
import time
import logging
import requests

logger = logging.getLogger(__name__)


class AlertRuleTester:
    def __init__(self, prom_url: str = "http://localhost:9090"):
        self.prom = prom_url.rstrip("/")

    def range_query(self, expr: str, minutes: int = 60, step: str = "30s"):
        # Replay an alert expr over recent history to see if it WOULD have fired.
        end = time.time()
        start = end - minutes * 60
        resp = requests.get(
            f"{self.prom}/api/v1/query_range",
            params={"query": expr, "start": start, "end": end, "step": step},
            timeout=15,
        )
        resp.raise_for_status()
        result = resp.json()["data"]["result"]
        return result

    def would_fire(self, expr: str, for_seconds: int, step_seconds: int = 30):
        # Emulate Prometheus `for:` — the expr must be truthy for a contiguous
        # run of samples at least for_seconds long before it transitions to firing.
        series = self.range_query(expr)
        if not series:
            logger.info("expr never truthy in window — alert stays silent")
            return False
        needed = for_seconds // step_seconds
        for s in series:
            run = 0
            for _, value in s["values"]:
                run = run + 1 if float(value) != 0 else 0
                if run >= needed:
                    logger.warning(
                        "alert WOULD fire on %s after %ss dwell",
                        s["metric"], for_seconds,
                    )
                    return True
        logger.info("truthy but never for a contiguous %ss — for: suppresses it", for_seconds)
        return False


if __name__ == "__main__":
    logging.basicConfig(level=logging.INFO)
    t = AlertRuleTester()
    # Synthetic lag: push a test metric via the Pushgateway, then validate the rule.
    # echo 'janusgraph_index_write_to_visible_seconds 45' | \
    #   curl --data-binary @- http://pushgateway:9091/metrics/job/synthetic_lag
    t.would_fire("janusgraph_index_write_to_visible_seconds > 30", for_seconds=300)

Implementation requirements:

  • The would_fire emulation of for: counts a contiguous truthy run, matching Prometheus semantics — a value that dips to zero resets the dwell, so intermittent breaches never satisfy a long for:. This is exactly why the harness catches a flapping expression before it ships.
  • Test-fire with the Pushgateway, not by editing thresholds down to meet current values. Push a synthetic janusgraph_index_write_to_visible_seconds at 45 seconds, confirm the rule fires, then delete the synthetic series so it does not linger and re-fire.
  • Run it in CI against a staging Prometheus so a threshold change is validated against real historical shape, not a hunch. The refresh-window arithmetic behind the index-lag threshold is derived step by step in Setting Alert Thresholds for Index Lag.

Diagnostics & Operational Fallbacks

Alerting itself fails in characteristic ways. Triage a noisy or silent pager against these symptom/diagnosis/resolution triplets.

  • Symptom: One incident produces five simultaneous pages — node down, pool saturated, lag high, aborts spiking, cache collapsing. Diagnosis: Missing inhibition. The downed-node root cause is not suppressing the downstream storage alerts it obviously causes. Resolution: Add the StorageNodeDown source inhibition scoped with equal: ['cluster'], verify the shortest for: is on the root-cause alert so it fires first, and confirm suppression in the Alertmanager UI’s inhibited list.

  • Symptom: An alert pages, clears, and re-pages every few minutes all night. Diagnosis: Flapping — the expression oscillates around the threshold and for: is too short or absent. Resolution: Add or lengthen for:, smooth the underlying gauge with avg_over_time(...[5m]), and raise repeat_interval so a genuinely persistent alert does not re-notify on the group cadence.

  • Symptom: Search returned stale results for twenty minutes but no page fired. Diagnosis: A missed page — the index-lag threshold is set above the lag the outage actually produced, or the metric stopped being scraped and the alert silently went stale. Resolution: Lower the static ceiling to match the refresh-window math, and add an absent(janusgraph_index_write_to_visible_seconds) alert so a missing metric pages rather than reads as healthy.

  • Symptom: The burn-rate SLO alert never fires even during a visible latency regression. Diagnosis: The recording rule feeding the ratio is broken or the SLO target is set so loose that no realistic error rate reaches the burn multiplier. Resolution: Verify the janusgraph:read_error_ratio:rateXm recording rules evaluate non-zero under load, tighten the SLO target to a value the service can actually miss, and test-fire with the Python harness above.

  • Symptom: Critical pages arrive batched several minutes late. Diagnosis: group_wait/group_interval on the critical route is inherited from the default and is batching pages that should be immediate. Resolution: Set group_wait: 0s on the critical route only, keep grouping for warnings, and confirm the route continue: false so a critical does not also fall through to the chat receiver.

For the upstream metric definitions behind these thresholds, see the Prometheus and JMX metrics reference, and for how these same signals are visualized before they page, see Grafana Dashboards.