Paging Rules for Storage-Backend Saturation

When a Cassandra or ScyllaDB backend saturates, the symptoms cascade outward — the CQL pool fills, native-transport requests queue, mutation stages block, GC pauses lengthen — and if each of those has its own page, one overloaded node wakes the whole rotation four times for a single root cause. This guide builds paging rules that fire on the earliest true saturation signal and suppress the derivative noise: it defines the four saturation signals worth paging on, sets thresholds that distinguish load from overload, encodes the Alertmanager routes and severities, and adds inhibition so a saturating node pages once, not once per downstream effect. It sits under Alerting Thresholds and Escalation and narrows that reference to the storage tier specifically. The failure it prevents is the page storm that trains on-call to bulk-silence, which is how the next saturation goes unanswered.

Deciding whether a storage saturation signal pages or is inhibited Four saturation signals are evaluated: CQL pool in-flight versus max, native-transport pending requests, tpstats blocked tasks, and GC pause time. Each crosses its own threshold and holds for a dwell window. A router then decides one of two outcomes per alert. If a root-cause saturation alert for the same host is already firing, the derivative alert is inhibited and suppressed so it does not page a second time. Otherwise the alert is routed to the pager as a critical with its runbook link, or to a ticket queue as a warning. The result is one page per saturation event rather than one page per downstream effect. SATURATION SIGNALS pool in-flight / max > 0.85 native-transport pending > 0 tpstats blocked rate > 0 GC pause time > 0.4 s/s threshold + for: dwell pending → firing root cause firing? yes inhibit — suppress no second page no · critical page + runbook pager, 0s group-wait warning ticket queue chat, business hours one page per saturation event, not one per downstream effect

Prerequisites

Confirm each item before writing paging rules — a saturation alert that fires on load rather than overload is as useless as one that never fires.

  • JanusGraph 0.6.x or 1.0.x on a CQL backend (Cassandra 3.11+/4.x or ScyllaDB), with the CQL driver pool metrics exported. The pool saturation signal is the ratio that Connection Pooling defines; you must know your configured max-connections-per-host to interpret it.
  • Cassandra/ScyllaDB metrics in Prometheus — either the Cassandra Exporter or a JMX exporter publishing nodetool tpstats equivalents (ThreadPools pending/blocked) and native-transport request counts.
  • JVM GC metrics (jvm_gc_pause_seconds or equivalent) scraped from the storage nodes.
  • A running Alertmanager whose routing tree you can edit and reload, plus a pager receiver (PagerDuty/OpsGenie or a test webhook).
  • Runbook URLs for each saturation class, reachable from the annotation link — a saturation page without a runbook is a page the responder cannot act on.
  • promtool and amtool on your path to validate rules and routing before reload.

Step 1 — Define the saturation signals

Saturation is not high utilization; it is queueing — work arriving faster than the backend drains it. Four signals capture it, ordered from earliest to most severe:

  • CQL pool in-flight vs max — janusgraph_cql_pool_in_flight / janusgraph_cql_pool_max_in_flight. The earliest predictor; it climbs before latency moves.
  • Native-transport pending — requests queued at the coordinator’s transport stage. Any sustained pending count means the node cannot accept work as fast as it arrives.
  • tpstats blocked — the rate of tasks the mutation/read thread pools have blocked. A non-zero blocked rate is unambiguous overload, not load.
  • GC pause time — pause seconds per second. Long pauses both cause and amplify saturation by stalling request draining.

Encode them as recording rules so every threshold reads from one canonical series:

yaml 
groups:
  - name: janusgraph-saturation-sli
    rules:
      - record: janusgraph:cql_pool_saturation:ratio
        expr: |
          janusgraph_cql_pool_in_flight
            / clamp_min(janusgraph_cql_pool_max_in_flight, 1)
      - record: cassandra:tpstats_blocked:rate5m
        expr: sum by (instance) (rate(cassandra_threadpool_blocked_tasks_total[5m]))
      - record: cassandra:gc_pause:ratio
        expr: sum by (instance) (rate(jvm_gc_pause_seconds_sum[5m]))

Verify: all three series resolve under normal load.

bash 
for q in janusgraph:cql_pool_saturation:ratio cassandra:tpstats_blocked:rate5m cassandra:gc_pause:ratio; do
  echo "== $q =="
  curl -s http://localhost:9090/api/v1/query --data-urlencode "query=$q" \
    | jq '.data.result[] | {instance: .metric.instance, v: .value[1]}'
done

Step 2 — Set paging thresholds

Distinguish load from overload. Pool saturation is normal up to a point, so page above 0.85 sustained; native-transport pending and tpstats blocked should be near zero in a healthy cluster, so any sustained non-zero value pages; GC pause pages above roughly 0.4 seconds of pause per second (40% of wall-clock spent in GC):

yaml 
groups:
  - name: janusgraph-saturation-paging
    rules:
      - alert: StoragePoolSaturated
        expr: janusgraph:cql_pool_saturation:ratio > 0.85
        for: 3m
        labels:
          severity: critical
          service: janusgraph
          component: storage
          saturation_class: pool
        annotations:
          summary: "CQL pool on {{ $labels.instance }} at {{ $value | humanizePercentage }}"
          runbook: "https://runbooks.internal/janusgraph/pool-saturation"
      - alert: StorageTransportPending
        expr: cassandra_native_transport_pending > 0
        for: 5m
        labels:
          severity: critical
          service: janusgraph
          component: storage
          saturation_class: transport
        annotations:
          summary: "Native-transport pending on {{ $labels.instance }}"
          runbook: "https://runbooks.internal/cassandra/transport-pending"
      - alert: StorageStageBlocked
        expr: cassandra:tpstats_blocked:rate5m > 0
        for: 2m
        labels:
          severity: critical
          service: janusgraph
          component: storage
          saturation_class: threadpool
        annotations:
          summary: "Blocked stage tasks on {{ $labels.instance }}"
          runbook: "https://runbooks.internal/cassandra/tpstats-blocked"
      - alert: StorageGCPauseHigh
        expr: cassandra:gc_pause:ratio > 0.4
        for: 3m
        labels:
          severity: warning
          service: janusgraph
          component: storage
          saturation_class: gc
        annotations:
          summary: "GC pause {{ $value | humanizePercentage }} of wall-clock on {{ $labels.instance }}"
          runbook: "https://runbooks.internal/cassandra/gc-pause"

The saturation_class label is the hook the inhibition rules use in Step 3. GC is a warning because it is usually a contributing cause rather than the actionable root — it should inform the pool or transport page, not add a third pager buzz.

Verify: the rule file parses.

bash 
promtool check rules janusgraph-saturation-paging.yml

Step 3 — Encode Alertmanager routes and inhibition

Route storage criticals straight to the pager, warnings to chat, and — the load-bearing part — inhibit the derivative saturation alerts on a host once its root-cause saturation is firing. A blocked thread pool is the deepest signal; when it fires, the pool and transport alerts for the same host are its symptoms and must not page again:

yaml 
route:
  receiver: default-chat
  group_by: ['service', 'instance', 'saturation_class']
  group_wait: 30s
  repeat_interval: 4h
  routes:
    - matchers:
        - service = "janusgraph"
        - severity = "critical"
      receiver: pagerduty-oncall
      group_wait: 0s
    - matchers:
        - service = "janusgraph"
        - severity = "warning"
      receiver: slack-graph-team

inhibit_rules:
  # A blocked thread pool is the root; suppress pool + transport pages on the same host
  - source_matchers:
      - alertname = "StorageStageBlocked"
    target_matchers:
      - saturation_class =~ "pool|transport"
    equal: ['instance']
  # High GC is a contributing cause; never let it page alongside a real saturation alert
  - source_matchers:
      - saturation_class = "pool"
    target_matchers:
      - saturation_class = "gc"
    equal: ['instance']

The equal: ['instance'] clause scopes every suppression to a single host, so a saturating node never silences a genuine page on a different node. This is the same source-then-suppress discipline the parent Alerting Thresholds and Escalation reference applies across the whole fleet.

Verify: the routing config is valid and a saturation alert routes where you expect.

bash 
amtool check-config alertmanager.yml
amtool config routes test \
  --config.file=alertmanager.yml \
  service=janusgraph severity=critical saturation_class=pool

Step 4 — Confirm the runbook link and end-to-end page

Every saturation alert must carry a runbook annotation that resolves, because the responder’s first click at 3 a.m. is that link. Verify the annotation renders with a real URL, then drive a synthetic page through the full path to confirm inhibition actually suppresses the derivative alerts.

bash 
# Confirm each firing alert carries a resolvable runbook annotation
curl -s http://localhost:9090/api/v1/alerts \
  | jq '.data.alerts[] | select(.labels.service=="janusgraph")
        | {alert: .labels.alertname, runbook: .annotations.runbook}'

Verify: with a blocked-stage alert firing, the pool and transport alerts for the same host appear as inhibited, not paging.

bash 
# Inhibited alerts are still "active" but suppressed — check Alertmanager's view
amtool alert query --alertmanager.url=http://localhost:9093 \
  service=janusgraph --active | grep -E "Suppressed|StoragePoolSaturated"

If the pool and transport alerts still page while the blocked-stage alert is firing, the equal label does not match — confirm both alerts actually carry an identical instance label value (exporter relabeling is the usual culprit).

Fallback and rollback procedures

Validate between actions rather than stacking changes.

If Step 2 thresholds page on normal load. Pool ratio briefly touching 0.85 during a burst is expected; that is what for: 3m absorbs. If it still pages, raise the dwell before the threshold — a threshold raised too high hides real saturation, whereas a longer dwell only delays a genuine one slightly.

If Step 3 inhibition does not suppress. The equal label mismatch is nearly always the cause. Compare the raw label sets of the source and target alerts; if the exporter emits instance as host:port on one series and host on another, add a relabel rule to normalize them before the inhibition can match.

If the whole ruleset over-pages after rollout. Roll back by restoring the previous Prometheus rule file and Alertmanager config from version control and reloading both (curl -XPOST :9090/-/reload and amtool reload). Re-introduce the alerts one saturation class at a time against a staging Alertmanager so you can see which class is noisy before it reaches the real pager.

If a saturation alert fires but the node is actually healthy. Suspect a stale or double-counted metric — a JMX exporter scraping a restarted node can report phantom pending counts. Cross-check against nodetool tpstats on the box directly before treating the page as real, and add an absent()-guarded staleness check if the exporter is prone to it.