Prometheus JMX Metrics

JanusGraph publishes almost everything an on-call engineer needs through JMX MBeans, and almost none of it reaches Prometheus without deliberate wiring. The JVM inside every Gremlin Server registers Dropwizard timers and gauges for storage latency, index dispatch, and cache behaviour, but those beans die inside the process unless an exporter scrapes the platform MBean server and reshapes their object names into label-carrying time series. This reference sits under the JanusGraph Monitoring and Observability reference and narrows it to one link in the chain: turning the raw javax.management object names JanusGraph emits into clean, alertable Prometheus metrics. The failure mode it prevents is silent blindness — the state where storage read latency has already tripled, the IndexProvider bulk queue is backing up, and the cache hit ratio has collapsed, yet every Grafana panel reads flat because the exporter dropped or mislabeled the beans that carry those signals. Everything below prioritizes an explicit allow/rename ruleset over the exporter’s default firehose, which will flood your time-series database with thousands of useless JVM internals and still miss the three MBeans that matter.

The diagram below traces a single metric from the JanusGraph JVM out to an alert: the platform MBean server holds the beans, the jmx_exporter agent reshapes and serves them on an HTTP port, Prometheus scrapes that port, and the resulting series fan out to Grafana panels and Alertmanager routes.

Core Configuration & Consistency Tuning

Three MBean groups carry the signals that predict a storage-backend incident before it pages you, and the exporter ruleset exists to keep exactly these while discarding the thousands of java.lang, java.nio, and TinkerPop internals that otherwise bloat the scrape. JanusGraph registers these as Dropwizard timers, meters, and counters under the org.janusgraph prefix; the JMX reporter publishes them into the platform MBean server where the agent can see them.

• org.janusgraph.diskstorage.cql.CQLStoreManager — the storage read/write path. Its getSlice and mutate timers carry per-operation latency percentiles, and its pool gauges report in-flight and available connections. This is the earliest saturation signal: getSlice p99 climbs the moment coordinator queues back up, well before NoHostAvailableException appears. Pool utilization here is the same ceiling discussed in Connection Pooling — when it pins at the configured maximum, threads are already blocking on acquisition.
• org.janusgraph.diskstorage.indexing.IndexProvider — the mixed-index dispatch path. Its queue-depth gauge and bulk-flush timer expose how far the Elasticsearch or OpenSearch write path lags behind storage commits. A rising queue with a flat flush rate is the definitive signature of index backpressure.
• org.janusgraph.graphdb.database.StandardJanusGraph — the transaction and cache layer. Its db.cache.retrievals and db.cache.misses counters yield the database-level cache hit ratio, and its tx.rollback (abort) meter versus tx.commit reveals contention: a rising abort rate means transactions are colliding on the same elements and replaying.

First enable JanusGraph’s own metrics subsystem and the JMX reporter, so the beans exist for the agent to read. Add this block to janusgraph.properties:

# Master switch — no beans are registered without this
metrics.enabled=true
metrics.prefix=org.janusgraph

# Publish Dropwizard metrics into the platform MBean server
metrics.jmx.enabled=true
metrics.jmx.domain=metrics
metrics.jmx.agentid=janusgraph

# Per-store and per-query granularity so labels are meaningful
metrics.merge-stores=false

Operational constraints, in the order they bite:

1. metrics.enabled=true is the master switch. With it off, JanusGraph registers no timers at all and the exporter serves an empty JanusGraph namespace regardless of how correct the ruleset is. This is the single most common cause of a “the exporter works but there are no janusgraph metrics” report.
2. metrics.jmx.enabled=true is separate from the master switch. JanusGraph can report metrics to CSV, Graphite, or Slf4j without ever touching JMX. The jmx_exporter reads only the platform MBean server, so this reporter must be on or the beans never appear as MBeans.
3. metrics.merge-stores=false preserves per-store labels. With merging on, edgestore, graphindex, and janusgraph_ids collapse into one aggregate timer and you lose the ability to tell which store is slow. Keep it off so the rename rules can attach a store label.
4. metrics.prefix must match your rename rules. If you change the prefix from org.janusgraph, every pattern regex below must change with it, or the rules match nothing and the beans pass through with their raw, unusable object names.

Now attach the jmx_exporter Java agent and give it an explicit allow/rename ruleset. The default configuration exports every bean on the MBean server; a production ruleset does the opposite — it names the beans you want and drops the rest. Write this to jmx_config.yaml:

startDelaySeconds: 0
lowercaseOutputName: true
lowercaseOutputLabelNames: true

# Allow-list: only these three JanusGraph bean groups reach Prometheus
includeObjectNames:
  - "metrics:name=org.janusgraph.stores.*,type=timers"
  - "metrics:name=org.janusgraph.storeManager.pool.*,type=gauges"
  - "metrics:name=org.janusgraph.index.*,type=*"
  - "metrics:name=org.janusgraph.db.cache.*,type=counters"
  - "metrics:name=org.janusgraph.tx.*,type=meters"

rules:
  # CQLStoreManager read/write latency — timer percentiles per store & op
  - pattern: 'metrics<type=timers, name=org\.janusgraph\.stores\.(\w+)\.(\w+)\.time><>(\d+)thPercentile'
    name: janusgraph_store_operation_latency_seconds
    valueFactor: 0.001
    labels:
      store: "$1"
      operation: "$2"
      quantile: "0.$3"
    type: GAUGE

  # CQLStoreManager pool utilization gauge
  - pattern: 'metrics<type=gauges, name=org\.janusgraph\.storeManager\.pool\.(\w+)><>Value'
    name: janusgraph_cql_pool_$1
    type: GAUGE

  # IndexProvider queue depth + bulk-flush timing
  - pattern: 'metrics<type=gauges, name=org\.janusgraph\.index\.(\w+)\.queue-size><>Value'
    name: janusgraph_index_queue_size
    labels:
      backend: "$1"
    type: GAUGE
  - pattern: 'metrics<type=timers, name=org\.janusgraph\.index\.(\w+)\.bulk-flush\.time><>(\d+)thPercentile'
    name: janusgraph_index_bulk_flush_seconds
    valueFactor: 0.001
    labels:
      backend: "$1"
      quantile: "0.$2"
    type: GAUGE

  # StandardJanusGraph cache counters + tx abort meter
  - pattern: 'metrics<type=counters, name=org\.janusgraph\.db\.cache\.(retrievals|misses)><>Count'
    name: janusgraph_db_cache_$1_total
    type: COUNTER
  - pattern: 'metrics<type=meters, name=org\.janusgraph\.tx\.(commit|rollback)><>Count'
    name: janusgraph_tx_$1_total
    type: COUNTER

  # Drop everything not matched above
  - pattern: ".*"

The trailing - pattern: ".*" with no name is the drop rule: any bean that survives the allow-list but matches no rename rule is discarded rather than exported raw. The second diagram shows how one rule collapses a verbose, un-queryable object name into a labelled series.

Index Synchronization Signals & Lag Metrics

The IndexProvider MBeans are the observability seam between the graph and its external search index, and they are the only place the sync lag becomes a number you can alert on. When mutations commit to storage faster than the mixed index can flush them, the queue-size gauge rises and the bulk-flush.time timer widens; together they quantify the index-synchronization window that governs whether a has() predicate returns freshly committed data.

Compute the effective lag as outstanding work divided by drain rate. If the queue holds $Q$ pending documents and the backend flushes bulk batches of size $B$ at a rate of $f$ flushes per second, the time to drain is:

$$t_{\text{drain}} = \frac{Q}{B \cdot f}$$

A drain time that trends upward across scrapes means writes are arriving faster than the index absorbs them — the same drift condition covered from the index side in OpenSearch Sync Patterns. Translate the raw MBeans into a live lag panel with PromQL:

# Pending documents waiting to be flushed to the mixed index
janusgraph_index_queue_size{backend="search"}

# p99 bulk-flush latency in seconds, per index backend
janusgraph_index_bulk_flush_seconds{backend="search", quantile="0.99"}

# Flush throughput — bulk operations completed per second
rate(janusgraph_index_bulk_flush_seconds_count{backend="search"}[5m])

Two constraints keep these signals trustworthy. First, metrics.merge-stores=false must remain set, because a merged view sums queue depth across backends and hides which index is drifting. Second, the exporter must scrape frequently enough to catch transient queue spikes — a 60-second scrape interval against a queue that fills and drains in 20 seconds will alias the spike to zero and report false health. The scrape-interval trade-off is why the alerting thresholds derived from these series need hysteresis rather than a bare comparison; that logic belongs downstream in the pages that consume these metrics.

Python Integration Pattern

Server-side JMX metrics tell you what the graph did; they cannot tell you what a client experienced, because queue time in the driver and network round-trips never touch a JanusGraph bean. A gremlin-python pipeline should therefore export its own client-side timers alongside the JMX scrape, so a latency regression can be attributed to the server, the network, or the client. The pattern below wraps every submit in a Prometheus histogram and exposes it on a second scrape port, giving Prometheus both halves of the latency budget.

import time
import logging
from gremlin_python.driver.client import Client
from prometheus_client import Histogram, Counter, Gauge, start_http_server

logger = logging.getLogger(__name__)

# Client-side series, scraped independently of the server's :9404 endpoint
QUERY_LATENCY = Histogram(
    "gremlin_client_query_seconds",
    "End-to-end submit latency observed by the client",
    labelnames=("operation",),
    buckets=(0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0),
)
QUERY_ERRORS = Counter(
    "gremlin_client_query_errors_total",
    "Client submit failures",
    labelnames=("operation", "kind"),
)
INFLIGHT = Gauge(
    "gremlin_client_inflight",
    "Submits currently awaiting a result",
)


class InstrumentedClient:
    def __init__(self, host: str, port: int = 8182, metrics_port: int = 9410):
        url = f"ws://{host}:{port}/gremlin"
        self.client = Client(url, "g")
        # A dedicated HTTP endpoint Prometheus scrapes for client-side series
        start_http_server(metrics_port)

    def submit(self, query: str, operation: str, bindings: dict | None = None):
        INFLIGHT.inc()
        start = time.perf_counter()
        try:
            result = self.client.submit(query, bindings or {}).all().result()
            QUERY_LATENCY.labels(operation=operation).observe(
                time.perf_counter() - start
            )
            return result
        except Exception as exc:
            # Label by exception type so timeouts and pool errors separate cleanly
            QUERY_ERRORS.labels(operation=operation, kind=type(exc).__name__).inc()
            logger.error("submit failed [%s]: %s", operation, exc)
            raise
        finally:
            INFLIGHT.dec()

    def close(self):
        self.client.close()

Implementation requirements:

• The histogram buckets must span your SLO boundary. If your target is p99 under 250 ms, include a 0.25 bucket edge, or histogram_quantile interpolates across a gap and reports a fictional percentile.
• Label errors by type(exc).__name__ so a NoHostAvailableException (pool exhaustion) separates from a GremlinServerError (query fault) in the gremlin_client_query_errors_total series — one points at the pool, the other at the traversal.
• The metrics_port (9410 here) is distinct from the agent’s 9404 so the two scrape jobs never collide, and the INFLIGHT gauge lets you correlate a client-side backlog against server-side janusgraph_cql_pool_* saturation in a single query.
• Subtracting the JMX janusgraph_store_operation_latency_seconds from gremlin_client_query_seconds isolates network and driver-queue time — the portion of the budget no server metric can attribute.

Scrape Configuration & Pipeline Operation

Prometheus pulls; it never receives a push from the exporter. Both endpoints — the JVM agent on 9404 and the client histogram on 9410 — are scrape targets Prometheus discovers and polls on an interval. Add both jobs to prometheus.yml:

scrape_configs:
  - job_name: janusgraph-jmx
    metrics_path: /metrics
    scrape_interval: 15s
    scrape_timeout: 10s
    static_configs:
      - targets:
          - gremlin-1:9404
          - gremlin-2:9404
          - gremlin-3:9404
        labels:
          tier: storage-backend
    relabel_configs:
      # Preserve the node identity as an `instance` label without the port
      - source_labels: [__address__]
        regex: "([^:]+):.*"
        target_label: node
        replacement: "$1"

  - job_name: gremlin-python-clients
    scrape_interval: 15s
    static_configs:
      - targets: ["ingest-worker-1:9410", "ingest-worker-2:9410"]

Operational rules for the scrape tier:

• Set scrape_timeout below scrape_interval. The agent reads the whole MBean server on each scrape; if a wide default ruleset makes that read slow, a timeout equal to the interval produces gaps that look like the target being down. A tight includeObjectNames allow-list keeps the read fast and the scrape well inside its timeout.
• Add a tier or node label at scrape time, not in every rule. Relabeling at the scrape job attaches topology labels uniformly, so a single alert expression covers the fleet without per-node duplication.
• Keep the two jobs on separate job_names. Mixing server and client series under one job makes it impossible to compute the network-time delta, because you lose the ability to select one side cleanly.
• Scrape every node, aggregate in PromQL. Point the job at all Gremlin Servers and let max by (quantile) or sum by (store) aggregate; scraping one node and assuming the fleet matches hides a single slow replica.

Once the series land, threshold breaches route to on-call through the same alerting path that carries schema violations — see Alert Routing for Violations for how those routes are grouped and silenced. The exact PromQL for percentiles and healthy ranges lives in the storage and index latency metrics reference.

Diagnostics & Operational Fallbacks

Metric pipelines fail quietly: the dashboard renders, the panels just read wrong. Triage against these symptom/diagnosis/resolution triplets, which target the failure modes specific to missing and mislabeled JMX series.

• Symptom: curl localhost:9404/metrics | grep janusgraph returns nothing, but JVM metrics like jvm_memory_bytes_used are present. Diagnosis: the agent is attached and healthy, but JanusGraph registered no beans — metrics.enabled or metrics.jmx.enabled is unset, so the org.janusgraph namespace does not exist on the MBean server. Resolution: set both metrics.enabled=true and metrics.jmx.enabled=true in janusgraph.properties, restart the Gremlin Server, and confirm the beans appear via jconsole before re-checking the exporter.

• Symptom: JanusGraph metrics appear but carry raw, dotted names like metrics_type_timers_name_org_janusgraph_stores_edgestore_getslice_time_99thpercentile. Diagnosis: the rename rules did not match, so the exporter fell through to its default name-mangling. Almost always the metrics.prefix was changed without updating the pattern regexes, or merge-stores altered the object-name shape. Resolution: align every rule pattern with the actual object name — read one raw bean from jconsole, paste its exact ObjectName into the regex, and confirm the capture groups line up with the store and operation segments.

• Symptom: the store label is missing or every series collapses to a single unlabeled timer. Diagnosis: metrics.merge-stores=true merged edgestore, graphindex, and the id store into one aggregate before the exporter ever saw them, so there is no per-store segment left to capture. Resolution: set metrics.merge-stores=false, restart, and re-scrape; the per-store beans reappear and the rename rule can attach the store label.

• Symptom: latency values look absurd — p99 of 47000 for a fast query. Diagnosis: unit mismatch. JanusGraph’s Dropwizard timers report milliseconds (or nanoseconds, by version), but the metric name ends in _seconds without a valueFactor to convert. Resolution: add valueFactor: 0.001 for millisecond timers (or 1e-9 for nanosecond timers) on the rule, so a metric named _seconds actually holds seconds and histogram_quantile math stays correct.

• Symptom: metrics vanish for one node during a Gremlin Server restart and never return, while other nodes report normally. Diagnosis: the agent -javaagent flag was dropped from that node’s JVM options, or its port 9404 is bound by another process after the restart, so Prometheus scrapes a dead target. Resolution: confirm the -javaagent:/opt/jmx_exporter/jmx_prometheus_javaagent.jar=9404:/opt/jmx_exporter/jmx_config.yaml argument is present in the node’s startup script, free the port, and restart; the step-by-step exporter guide has the exact attach sequence and per-step verification.

For the upstream reference on how JanusGraph registers these timers, see the Apache TinkerPop Gremlin Server documentation on metric reporters.

Related

• Up a level: JanusGraph Monitoring and Observability — the observability reference this metrics pipeline feeds.
• Exporting JanusGraph JMX Metrics to Prometheus — the step-by-step agent attach, YAML, and verification procedure.
• Storage and Index Latency Metrics Reference — the metric-by-metric table with healthy ranges and p99 PromQL.
• Connection Pooling — the pool whose utilization the CQLStoreManager gauges report.
• OpenSearch Sync Patterns — the index-drift condition the IndexProvider queue metrics quantify.
• Alert Routing for Violations — the on-call routing these threshold breaches flow into.