JanusGraph Monitoring & Observability

Every failure mode on this site — a starved connection pool, a drifting mixed index, a saturated storage coordinator, a schema violation slipping into production — shares one property: it is invisible until you have instrumented for it, and by the time it surfaces in a user-facing traversal the incident is already minutes old. Monitoring is therefore not a reporting afterthought on a JanusGraph deployment; it is the control loop that makes the storage tier’s tunable guarantees and the index tier’s inherent lag observable before they become an outage. This reference is the entry point for the observability tier. It sits alongside the JanusGraph Storage Backend Architecture & Configuration, External Index Synchronization & Consistency Tuning, and Graph Schema Validation & Modeling Strategies references, and links down into every metrics-, dashboard-, and alerting-specific guide. The organizing principle throughout is that a signal you cannot graph and cannot page on does not exist operationally, no matter how carefully the underlying subsystem was tuned.

The diagram below shows how measurements leave the three JanusGraph tiers, land in a time-series store, and split into the two consumers that matter on call — dashboards for diagnosis and alerts for escalation.

Core Observability Model & Signal Boundaries

Observability on a JanusGraph cluster is only useful if it is organized around the same tier boundaries that produce incidents. A flat wall of graphs hides the one fact that matters during a page: which layer is actually failing. Group every signal into one of three classes, because each has a different owner, a different remediation, and a different urgency.

1. Storage-tier signals — CQL read/write latency percentiles, coordinator queue depth, native-transport pending tasks, compaction backlog, and connection-pool utilization. These originate in Cassandra or ScyllaDB and in the DataStax driver underneath JanusGraph. They fail loudly: writes reject, nodetool status shows down nodes, and the latency histograms move within one scrape interval. The remediation is capacity, topology, or pool sizing.
2. Index-tier signals — the IndexProvider mutation-queue depth, Elasticsearch/OpenSearch bulk-queue depth, refresh duration, and the derived index lag: the interval between a durable storage commit and the moment that mutation is visible to a has() predicate. These fail quietly — traversals keep returning, but they return stale or incomplete results. The remediation is refresh/bulk tuning or a reindex.
3. Compute-tier signals — database-cache hit ratio, transaction-abort rate, Gremlin Server thread-pool saturation, and JVM heap/GC behavior. These sit between the other two and often precede a visible incident: a collapsing cache hit ratio or lengthening GC pauses will show up minutes before p99 traversal latency does.

The single most important derived metric on the whole cluster is index lag, because it quantifies the non-atomic seam between synchronous storage commits and asynchronous index dispatch that the Eventual vs Strong Consistency reference develops in depth. Every other signal tells you a subsystem is unhealthy; index lag tells you the graph is lying to queries while every subsystem still reports green. Instrument it first, alert on it first, and put it at the top of every dashboard.

Where each signal is emitted

JanusGraph publishes its internal timers and counters through JMX when metrics are enabled, and the storage and index backends publish their own metrics endpoints independently. The observability tier’s job is to unify these three sources into one time-series store so a single PromQL expression can correlate a storage-latency spike with an index-lag climb. The wiring that does the unification — a Prometheus & JMX metrics catalog on the JanusGraph side, plus the backends’ native exporters — is the foundation everything else on this reference builds on.

Production Metrics Configuration

JanusGraph’s metrics subsystem is off by default. Turn it on explicitly and register the JMX reporter so a Prometheus JMX exporter can scrape it; nothing downstream works until these are set. The block below is a hardened baseline for a production Gremlin Server, with every non-default value justified beneath it.

# Metrics core
metrics.enabled=true
metrics.prefix=janusgraph

# JMX reporter — the surface the jmx_exporter agent reads
metrics.jmx.enabled=true
metrics.jmx.domain=org.janusgraph

# Optional direct Graphite/CSV reporters stay off; JMX -> Prometheus is the path
metrics.graphite.enabled=false

# Storage + index instrumentation depth
storage.cql.metrics.enabled=true
index.search.enabled=true

Rationale for every non-default value:

• metrics.enabled=true — the master switch. With it off, mgmt timers, the StandardJanusGraph cache beans, and CQL store timers are never registered and JMX exposes nothing useful. There is no partial mode; leave it on in every environment so staging reproduces production’s signals.
• metrics.prefix=janusgraph — pins a stable metric namespace. Without a fixed prefix the bean names embed a per-instance graph name, which fragments your Prometheus series across restarts and breaks every dashboard query. Set it once and never change it, because renaming it orphans historical data.
• metrics.jmx.enabled=true / metrics.jmx.domain — exposes the registry over JMX under a predictable domain, which is exactly what the exporter’s allow-rules match on. This is the seam between JanusGraph and the JMX-to-Prometheus exporter.
• storage.cql.metrics.enabled=true — turns on the DataStax driver’s node- and session-level metrics (in-flight requests, CQL latency, errors). These are the earliest warning of the connection-pool saturation covered under Connection Pooling; without them, pool exhaustion is invisible until it throws.

Enabling metrics has a non-zero cost: histogram reservoirs and JMX registration consume heap and a little CPU. That cost is trivial next to running blind, but size the JVM with it in mind and confirm the metrics registry is not competing with an oversized cache.db-cache-size for heap — the sizing interplay is worked through under Storage Backend Selection & Capacity Sizing.

The Metric Catalog: MBeans, Backends & Index Lag

Three JanusGraph JMX MBeans carry the signals worth paging on. Expose these at minimum, then add the storage- and index-backend native metrics beside them so one dashboard correlates all three tiers.

• org.janusgraph.diskstorage.cql.CQLStoreManager — CQL read/write latency percentiles and connection-pool utilization. Rising pool utilization with flat throughput is the signature of a starved pool; pair it with the driver’s own cql_requests and pool_in_flight gauges.
• org.janusgraph.diskstorage.indexing.IndexProvider — the mutation-queue size and bulk-flush duration. This is the index-drift early-warning system: alert when queue depth trends upward across successive scrapes rather than on a single spike.
• org.janusgraph.graphdb.database.StandardJanusGraph — cache hit/miss ratios and transaction-abort rate. A collapsing cache hit ratio usually precedes a latency incident by minutes, which is exactly the lead time a good alert needs.

Index lag is not published directly — you compose it. The visible window a query can be stale for is the sum of the time a mutation waits in the JanusGraph queue, the bulk transport to the search backend, and the backend’s own refresh interval:

$$t_{\text{lag}} = t_{\text{queue}} + t_{\text{bulk}} + t_{\text{refresh}}$$

Each term maps to a metric you already expose: t_queue from the IndexProvider queue depth divided by drain rate, t_bulk from the Elasticsearch/OpenSearch _cat/thread_pool/write bulk latency, and t_refresh from the configured refresh_interval. Graphing the composed value against your read-after-write requirement is the single most valuable panel you can build; the storage-and-index latency metrics reference enumerates the exact series and PromQL, and the Grafana dashboards reference turns them into panels.

Python Pipeline Observability

Server-side JMX metrics tell you how JanusGraph is behaving; they say nothing about how your ingestion clients experience it. A gremlin-python pipeline needs its own client-side timers so you can attribute a latency incident to the driver, the network, or the server. The pattern below wraps every submission in a timer, counts outcomes by result, and exposes them to the same Prometheus that scrapes the server — closing the loop between client-observed and server-reported latency.

import time
import logging
from prometheus_client import Counter, Histogram, start_http_server
from gremlin_python.driver.driver_remote_connection import DriverRemoteConnection
from gremlin_python.process.anonymous_traversal import traversal

logger = logging.getLogger(__name__)

# Client-side series, scraped on :9410 alongside the server's :9404.
SUBMIT_LATENCY = Histogram(
    "gremlin_submit_seconds",
    "Client-observed traversal submit latency",
    buckets=(0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0),
)
SUBMIT_RESULT = Counter(
    "gremlin_submit_total",
    "Traversal submissions by outcome",
    ["outcome"],
)


class ObservedGraph:
    def __init__(self, url: str):
        self._conn = DriverRemoteConnection(url, "g")
        self.g = traversal().with_remote(self._conn)

    def timed(self, fn):
        # One timer + one labelled counter per call; failures are counted, not
        # swallowed, so a rising error rate is visible before it becomes a page.
        start = time.perf_counter()
        try:
            result = fn(self.g)
            SUBMIT_RESULT.labels(outcome="ok").inc()
            return result
        except Exception:
            SUBMIT_RESULT.labels(outcome="error").inc()
            logger.exception("traversal failed")
            raise
        finally:
            SUBMIT_LATENCY.observe(time.perf_counter() - start)

    def close(self):
        self._conn.close()


if __name__ == "__main__":
    start_http_server(9410)  # expose client metrics for Prometheus
    graph = ObservedGraph("ws://janusgraph-server:8182/gremlin")
    try:
        count = graph.timed(lambda g: g.V().count().next())
        logger.info("vertex count = %s", count)
    finally:
        graph.close()

Client-instrumentation rules that hold across every pipeline:

• Count failures, never swallow them. The error label on SUBMIT_RESULT is what turns a silent retry storm into a graphable rate. A pipeline that catches and hides exceptions is indistinguishable from a healthy one until it falls over.
• Histogram, not gauge, for latency. A histogram lets Prometheus compute any percentile after the fact; a single gauge of “last latency” is noise. Size the buckets around your SLO so the bucket boundaries straddle the threshold you alert on.
• Expose client metrics on a separate port and scrape both. The gap between client-observed gremlin_submit_seconds p99 and the server’s CQL latency is your network-and-driver overhead — an essential term when triaging whether a slowdown is the graph or the path to it.

For the transaction-boundary and idempotency discipline these pipelines depend on, work from the ingestion pattern in Bulk Data Loading; observability rides on top of correct transaction handling, it does not replace it.

Diagnostics, Dashboards & Escalation

The three child references turn the model above into running infrastructure. Stand them up in order: a Prometheus & JMX metrics catalog to get the series flowing, Grafana dashboards to make them legible during an incident, and alerting thresholds to page a human before the incident reaches a user. Wire the schema-integrity alerts from Alert Routing for Violations into the same Alertmanager so operational and data-quality pages share one escalation path.

Failure-mode reference

The failure modes below account for the majority of “we were blind to it” observability gaps. Each maps a symptom to the query or command that confirms the diagnosis and the resolution that closes the gap.

Related

• Alongside: Storage Backend Architecture & Configuration, External Index Synchronization & Consistency Tuning, and Graph Schema Validation & Modeling Strategies — the three subsystems this observability tier instruments.
• Prometheus & JMX Metrics — exposing JanusGraph’s JMX MBeans as Prometheus series and the metric catalog to keep.
• Grafana Dashboards — panels and PromQL for storage latency, index lag, pool saturation, and cache health.
• Alerting Thresholds — turning signals into pages that fire before an incident reaches a user, without alert fatigue.
• Eventual vs Strong Consistency — the storage-versus-index consistency model that index lag quantifies.
• Connection Pooling — the pool-saturation signals to expose before they throw.