Routing Schema-Violation Alerts to On-Call

This guide is the concrete wiring that turns a JanusGraph schema violation into a page an engineer can act on, using the standard Prometheus and Alertmanager stack rather than a bespoke dispatcher. It sits under the Alert Routing for Violations reference, which defines the severity model this procedure implements, and it narrows that model to one repeatable task: emit a signal from the validation layer, scrape it, evaluate it, and route it to on-call with the deduplication and hold-down that keep a real SCHEMA_VIOLATION from drowning in transient noise. The specific failure it prevents is the silent one — a malformed write that commits, corrupts a query result days later, and never once touched a pager because nothing along the path ever emitted a metric a rule could fire on.

Schema-violation signal path from the validation layer through Prometheus and Alertmanager to on-call receivers The validation layer increments a schema_violations_total counter and writes a structured log. Prometheus scrapes the counter on a fixed interval, evaluates an alerting rule, and holds the alert pending for a for: duration before it becomes firing. The firing alert is sent to Alertmanager, which groups and deduplicates it and routes by the severity label: a critical alert pages PagerDuty or Opsgenie on-call, while a warning goes only to a Slack channel. Validation layer counter + structured log schema_violations_total Prometheus scrape + evaluate rule for: 2m hold Alertmanager group · dedup route by severity critical PagerDuty · Opsgenie warning Slack channel scrape fires
The validation layer emits a counter and log; Prometheus scrapes and holds the rule pending for a for: window; Alertmanager deduplicates and fans out by severity — critical pages on-call while a warning stays in Slack.

Prerequisites

Confirm each item before touching alerting config. The most common way this wiring fails is a metric that never leaves the JanusGraph process, so the emit step and the scrape step must both be proven independently.

  • A JanusGraph 0.6.x or 1.0.x deployment with a validation layer you control — the client-edge check described in Vertex and Edge Validation, or a server-side interceptor around graph.tx().commit(). You need a code seam where a rejected mutation is observable.
  • prometheus_client (Python) installed in the process that performs validation, and a reachable HTTP port for its exposition endpoint.
  • Prometheus 2.40+ and Alertmanager 0.25+ already scraping your fleet. If you are still standing up the metrics pipeline, wire the exporter first per Prometheus & JMX Metrics and confirm targets are up before adding rules.
  • A receiver integration key for each destination: a PagerDuty Events API v2 routing key or an Opsgenie API key for critical, and a Slack incoming-webhook URL for warning. Store them as secrets, not in the committed YAML.
  • Agreed thresholds. Decide, with the numbers from Alerting Thresholds, what rate of violations is a page versus a Slack note. A single stray SCHEMA_VIOLATION may warrant a page; a slow trickle of INDEX_SYNC_LAG never should.

Step 1 — Emit a schema-violation signal from the validation layer

The router can only route what the graph makes observable. Add a counter and a structured log to the exact point where a mutation is rejected, labelled so a rule can slice by violation type and severity. Increment the counter before re-raising, so the signal survives even if the caller swallows the exception.

python 
import logging
from prometheus_client import Counter, start_http_server

logger = logging.getLogger("janusgraph.validation")

# Cardinality-safe labels only: bounded type/label/severity, never the payload.
SCHEMA_VIOLATIONS = Counter(
    "janusgraph_schema_violations_total",
    "Count of rejected JanusGraph mutations by violation type and severity",
    ["violation_type", "vertex_label", "severity"],
)

SEVERITY = {
    "SCHEMA_VIOLATION": "critical",
    "QUORUM_UNAVAILABLE": "critical",
    "INDEX_SYNC_STALL": "warning",
    "INDEX_SYNC_LAG": "warning",
}


def record_violation(violation_type: str, vertex_label: str, detail: str) -> None:
    severity = SEVERITY.get(violation_type, "warning")
    SCHEMA_VIOLATIONS.labels(violation_type, vertex_label, severity).inc()
    # Structured log carries the un-aggregatable detail the metric must not.
    logger.warning(
        "schema_violation",
        extra={"violation_type": violation_type, "vertex_label": vertex_label,
               "severity": severity, "detail": detail},
    )


def validate_mutation(mutation: dict) -> None:
    if "label" not in mutation or "properties" not in mutation:
        record_violation("SCHEMA_VIOLATION", mutation.get("label", "unknown"),
                         "missing label/properties")
        raise ValueError("Missing required mutation fields")


if __name__ == "__main__":
    start_http_server(9101)   # exposes /metrics on :9101 for the scrape

Keep label cardinality bounded: violation_type, vertex_label, and severity are all small closed sets. Never put the offending value, vertex id, or free-text detail on a metric label — that belongs in the structured log, and doing otherwise blows up Prometheus memory.

Verify: confirm the counter is exposed and increments on a rejected write.

bash 
curl -s http://localhost:9101/metrics | grep janusgraph_schema_violations_total
# Expect a HELP/TYPE header now, and a non-zero sample after a bad mutation:
# janusgraph_schema_violations_total{severity="critical",violation_type="SCHEMA_VIOLATION",vertex_label="unknown"} 1.0

Step 2 — Scrape the metric into Prometheus

Add a scrape job so Prometheus pulls the exposition endpoint on a fixed interval. The scrape interval is the first term in your detection budget, so keep it tight for the validation endpoint even if the rest of the fleet scrapes more slowly.

yaml 
# prometheus.yml (scrape_configs excerpt)
scrape_configs:
  - job_name: "janusgraph-validation"
    scrape_interval: 15s
    static_configs:
      - targets: ["graph-node-01.internal:9101", "graph-node-02.internal:9101"]
        labels:
          service: "janusgraph"
          cluster: "graph-prod"

The end-to-end time from a violation to a page is the sum of three bounded terms — the scrape that ingests the sample, the for: window that suppresses flapping, and the Alertmanager group-wait that batches the notification:

Tdetect≤tscrape+tfor+tgroup_waitT_{\text{detect}} \le t_{\text{scrape}} + t_{\text{for}} + t_{\text{group\_wait}}

With t_scrape = 15s, t_for = 2m, and t_group_wait = 30s, worst-case detection stays under three minutes. Size each term against your paging SLA rather than copying these values blindly.

Verify: confirm the target is up and the series is queryable.

bash 
# Target health
curl -s 'http://prometheus:9090/api/v1/targets' \
  | jq '.data.activeTargets[] | select(.labels.job=="janusgraph-validation") | .health'
# Series is present
curl -s 'http://prometheus:9090/api/v1/query' \
  --data-urlencode 'query=janusgraph_schema_violations_total' | jq '.data.result | length'

Step 3 — Write the alerting rule

Translate the counter into a firing condition. Alert on the rate over a short window, not the raw total, so a counter reset on restart does not fabricate a spike. Carry the severity label straight through from the metric into the alert so Alertmanager can route on it, and set for: so a single transient sample does not page.

yaml 
# rules/janusgraph-schema.yml
groups:
  - name: janusgraph-schema-violations
    rules:
      - alert: JanusGraphSchemaViolationCritical
        expr: |
          sum by (violation_type, vertex_label) (
            rate(janusgraph_schema_violations_total{severity="critical"}[5m])
          ) > 0
        for: 2m
        labels:
          severity: critical
          team: graph-oncall
        annotations:
          summary: "Critical schema violation on {{ $labels.vertex_label }}"
          description: "{{ $labels.violation_type }} rejected on {{ $labels.vertex_label }} — check structured logs for the payload."
      - alert: JanusGraphSchemaViolationWarning
        expr: |
          sum by (violation_type) (
            rate(janusgraph_schema_violations_total{severity="warning"}[5m])
          ) > 0.05
        for: 10m
        labels:
          severity: warning
          team: graph-oncall
        annotations:
          summary: "Elevated warning-level graph violations"
          description: "{{ $labels.violation_type }} above the warning rate for 10m."

The critical rule fires on any sustained critical violation; the warning rule tolerates a low trickle and only fires above a rate floor, holding for a longer for: because it is not worth waking anyone.

Verify: lint the rule file and confirm it loads.

bash 
promtool check rules rules/janusgraph-schema.yml
# After reload, confirm Prometheus registered the alert:
curl -s 'http://prometheus:9090/api/v1/rules' \
  | jq '.data.groups[].rules[] | select(.name|startswith("JanusGraphSchema")) | .name'

Step 4 — Route through Alertmanager to on-call

Alertmanager decides where a firing alert lands, collapses duplicates, and controls re-notification cadence. Group by the labels that define one incident so a burst of identical violations becomes a single notification, and branch on severity so critical reaches the pager while warning stays in Slack.

yaml 
# alertmanager.yml
route:
  receiver: graph-slack                 # default sink for anything unmatched
  group_by: ["alertname", "violation_type", "vertex_label"]
  group_wait: 30s                       # batch the first burst
  group_interval: 5m                    # cadence for new members of a group
  repeat_interval: 4h                   # re-page cadence while still firing
  routes:
    - matchers: ['severity="critical"']
      receiver: graph-pagerduty
      continue: false
    - matchers: ['severity="warning"']
      receiver: graph-slack

receivers:
  - name: graph-pagerduty
    pagerduty_configs:
      - routing_key_file: /etc/alertmanager/secrets/pagerduty_key
        severity: critical
    # Swap for opsgenie_configs with an api_key_file if Opsgenie is your pager.
  - name: graph-slack
    slack_configs:
      - api_url_file: /etc/alertmanager/secrets/slack_webhook
        channel: "#graph-violations"
        title: "{{ .CommonAnnotations.summary }}"
        text: "{{ .CommonAnnotations.description }}"

inhibit_rules:
  - source_matchers: ['severity="critical"']
    target_matchers: ['severity="warning"']
    equal: ["violation_type", "vertex_label"]

Grouping is what stops a pager storm: a single bad producer emitting thousands of identical SCHEMA_VIOLATION events collapses to one grouped notification plus a count. The inhibit_rules block suppresses the warning-level alert for a (violation_type, vertex_label) pair whenever the critical alert for the same pair is already firing, so on-call is not double-notified for one root cause.

Verify: validate the config and confirm the route resolves to the right receiver.

bash 
amtool check-config alertmanager.yml
amtool config routes test --config.file=alertmanager.yml severity=critical
# Expect: graph-pagerduty
amtool config routes test --config.file=alertmanager.yml severity=warning
# Expect: graph-slack

Step 5 — Test-fire the alert end to end

Do not wait for a real violation to discover the routing key is wrong. Inject a synthetic alert straight into Alertmanager and confirm it reaches the pager, then remove it.

bash 
# Fire a synthetic critical alert with a short expiry
amtool alert add \
  alertname=JanusGraphSchemaViolationCritical \
  severity=critical violation_type=SCHEMA_VIOLATION vertex_label=account \
  --annotation=summary="TEST — synthetic schema violation" \
  --end="$(date -u -d '+5 minutes' +%Y-%m-%dT%H:%M:%SZ)" \
  --alertmanager.url=http://alertmanager:9093

For a full-path test that also exercises the rule, push the counter on a scratch instance and let Prometheus evaluate it naturally — that proves the for: hold and the scrape interval, not just the last hop.

Verify: confirm the alert is active and the receiver acknowledged it.

bash 
amtool alert query alertname=JanusGraphSchemaViolationCritical \
  --alertmanager.url=http://alertmanager:9093
# In PagerDuty/Opsgenie: confirm an incident was created on the graph-oncall service.
# Then resolve the synthetic alert so it does not linger:
amtool silence add alertname=JanusGraphSchemaViolationCritical \
  --duration=1h --comment="clear synthetic test" \
  --alertmanager.url=http://alertmanager:9093

Fallback and rollback procedures

Each step has an isolated recovery path. Validate between actions instead of stacking changes on top of a broken hop.

If Step 1 emits nothing. The counter is registered lazily, so a label combination is absent until it first increments — this is expected. If /metrics itself is empty or unreachable, the exposition server never started; confirm start_http_server runs in the live process (not a forked worker) and that the port is not firewalled from Prometheus. Roll back by reverting the instrumentation commit; the graph keeps rejecting bad writes regardless, you have only lost the signal.

If Step 2 shows the target down. Prometheus cannot reach the endpoint. Check network policy between the Prometheus host and port 9101, and confirm the targets host matches the pod or node the process actually runs on. Until it is fixed, no rule can fire — treat a down validation target as itself alertable via a up{job="janusgraph-validation"} == 0 rule so a blind spot pages rather than hides.

If Step 3 rule never fires. Query the expr directly in the Prometheus UI. A common cause is a rate() window shorter than the scrape interval, which yields no data points; keep the range at least four scrape intervals wide. If it fires too eagerly, raise the rate floor or lengthen for: rather than deleting the rule.

If Step 4 routes to the wrong place. amtool config routes test is the authority — if it disagrees with reality, the running Alertmanager has stale config. Reload it (curl -XPOST http://alertmanager:9093/-/reload) and re-test. To roll back a misrouted change, restore the previous alertmanager.yml, reload, and re-run the route test before declaring it fixed.

If Step 5 never pages. The synthetic alert reaching amtool alert query but not the pager isolates the fault to the receiver integration — a wrong routing key or a revoked Slack webhook. Rotate the secret, reload, and re-fire. Never leave a silence in place after testing; an orphaned silence is how a real page gets swallowed.

Frequently Asked Questions

Should the alert fire on the raw counter total or on its rate? On the rate. janusgraph_schema_violations_total is a monotonic counter that resets to zero on process restart, so a rule that reads the absolute value fires spuriously after every deploy. Use rate(...[5m]) > 0 with a range at least four scrape intervals wide so a counter reset never looks like a new violation spike.

Why put a for: on the rule instead of paging immediately? The for: clause makes Prometheus hold the alert in a pending state until the condition has been continuously true for that duration, which suppresses a single flapping sample. For a hard SCHEMA_VIOLATION a short 2m hold is enough to filter noise while staying inside a paging SLA; a warning-level rule takes a longer hold because it is never worth waking someone for.

How do I stop one bad producer from causing a pager storm? Alertmanager grouping. By setting group_by to the labels that define a single incident — alertname, violation_type, vertex_label — thousands of identical events collapse into one notification with an occurrence count, and group_interval plus repeat_interval bound how often it re-notifies while still firing.

Can I route to Opsgenie or Slack instead of PagerDuty without rewriting the rule? Yes. The rule only sets the severity label; the destination is entirely an Alertmanager receiver concern. Swap pagerduty_configs for opsgenie_configs with an api_key_file, or point the critical route at a Slack receiver, and the rule and the emitting code are untouched.