Exporting JanusGraph JMX Metrics to Prometheus

This guide is the exact sequence for attaching the jmx_exporter Java agent to a running Gremlin Server, shaping its output down to the JanusGraph beans that matter, and confirming the metrics land in Prometheus — so you never discover mid-incident that the storage-latency panel was empty the whole time. It sits under the Prometheus JMX Metrics reference and narrows that subsystem to a single deployable task: wiring one node from a JVM with no exporter to a verified scrape target returning janusgraph_* series. The specific failure this prevents is the false-negative dashboard — panels that render flat green because the agent was attached without a ruleset, or with a ruleset whose regexes never matched the live object names. Every step below ends in a command that either proves the change worked or tells you precisely which layer broke.

The agent-attach sequence from JVM flag to verified Prometheus target Five ordered stages carry a node from unmonitored to scraped. First the operator adds a javaagent JVM flag pointing at the jmx_prometheus_javaagent jar, a port, and the config YAML. On Gremlin Server restart the agent loads inside the JVM and reads the platform MBean server. The agent applies the keep-and-rename ruleset and binds an HTTP listener on port 9404. A curl to localhost 9404 slash metrics returns renamed janusgraph series, confirming the exporter side. Finally Prometheus, configured with a scrape job for the node on 9404, polls the endpoint on its interval and the target shows up in the Prometheus targets page. Each stage has a verification command so a break is localized to one layer. 1 · JVM flag -javaagent=jar :9404:config.yaml 2 · agent loads reads MBean server on restart 3 · rename keep janusgraph bind :9404 4 · curl grep janusgraph series present 5 · scrape Prometheus job target UP each stage ends in a Verify command — a break localizes to one layer operator edits Prometheus pulls
The attach sequence: a JVM flag loads the agent, the agent renames and serves on 9404, curl proves the exporter, and a scrape job promotes the node to a live Prometheus target.

Prerequisites

Confirm each item before touching the Gremlin Server startup script. Skipping the version and path checks is the most common reason an agent attaches but serves nothing.

  • JanusGraph 0.6.x or 1.0.x running on a reachable Gremlin Server, with metrics.enabled=true and metrics.jmx.enabled=true already set in janusgraph.properties. If those switches are off, the Prometheus JMX Metrics reference covers the full metrics.* block to enable first.
  • The jmx_exporter Java agent jar — jmx_prometheus_javaagent-1.0.1.jar or newer — downloaded to a stable path such as /opt/jmx_exporter/ on every Gremlin Server node.
  • A free TCP port (9404 by convention) on each node, and firewall or security-group rules that let the Prometheus server reach it.
  • Write access to the Gremlin Server startup script (bin/gremlin-server.sh or the systemd unit’s Environment=JAVA_OPTIONS) and a maintenance window to restart the JVM — the agent attaches only at process start.
  • A running Prometheus you can edit prometheus.yml on and reload, plus curl and jq on the node for verification.

Step 1 — Add the javaagent flag to Gremlin Server

The agent must be on the JVM command line before the process starts; it cannot be attached to a live JVM cleanly. Export it through JAVA_OPTIONS, which Gremlin Server’s launcher appends to the java invocation. The flag’s three colon-separated parts are the jar, the listen port, and the config path.

bash 
# In the Gremlin Server environment (systemd unit or gremlin-server.sh)
export JAVA_OPTIONS="$JAVA_OPTIONS \
  -javaagent:/opt/jmx_exporter/jmx_prometheus_javaagent-1.0.1.jar=9404:/opt/jmx_exporter/jmx_config.yaml"

Do not start the server yet — the config path points at a file you write in Step 2. Staging the flag first keeps the restart in Step 3 a single, atomic action.

Verify: the flag is present and the jar exists at the referenced path.

bash 
echo "$JAVA_OPTIONS" | grep -o "javaagent:[^ ]*"
test -f /opt/jmx_exporter/jmx_prometheus_javaagent-1.0.1.jar && echo "jar OK" || echo "jar MISSING"

Step 2 — Write the keep-and-rename exporter YAML

An empty config makes the agent export every bean on the MBean server — thousands of java.lang internals that bury the JanusGraph series and slow each scrape. Write a ruleset that keeps only the storage, index, and cache beans and renames them to clean metric names with labels.

yaml 
# /opt/jmx_exporter/jmx_config.yaml
lowercaseOutputName: true
lowercaseOutputLabelNames: true

includeObjectNames:
  - "metrics:name=org.janusgraph.stores.*,type=timers"
  - "metrics:name=org.janusgraph.index.*,type=*"
  - "metrics:name=org.janusgraph.db.cache.*,type=counters"
  - "metrics:name=org.janusgraph.tx.*,type=meters"

rules:
  # Storage read/write latency per store and operation, ms -> s
  - pattern: 'metrics<type=timers, name=org\.janusgraph\.stores\.(\w+)\.(\w+)\.time><>(\d+)thPercentile'
    name: janusgraph_store_operation_latency_seconds
    valueFactor: 0.001
    labels: {store: "$1", operation: "$2", quantile: "0.$3"}
    type: GAUGE
  # Mixed-index queue depth
  - pattern: 'metrics<type=gauges, name=org\.janusgraph\.index\.(\w+)\.queue-size><>Value'
    name: janusgraph_index_queue_size
    labels: {backend: "$1"}
    type: GAUGE
  # Cache retrievals and misses for the hit-ratio calculation
  - pattern: 'metrics<type=counters, name=org\.janusgraph\.db\.cache\.(retrievals|misses)><>Count'
    name: janusgraph_db_cache_$1_total
    type: COUNTER
  # Transaction commit and abort meters
  - pattern: 'metrics<type=meters, name=org\.janusgraph\.tx\.(commit|rollback)><>Count'
    name: janusgraph_tx_$1_total
    type: COUNTER
  # Drop anything unmatched so the scrape stays lean
  - pattern: ".*"

The final unnamed - pattern: ".*" is the drop rule: it silently discards any bean that survived the include list but matched no rename, so a stray attribute never leaks out with a mangled name.

Verify: the YAML parses. A tab character or a bad indent here fails the agent silently at load.

bash 
python3 -c "import yaml,sys; yaml.safe_load(open('/opt/jmx_exporter/jmx_config.yaml')); print('yaml OK')"

Step 3 — Restart Gremlin Server and open the port

Restart the JVM so the staged agent loads and binds its listener. Drain in-flight traffic first if this node takes production writes.

bash 
# systemd example — adjust to your service manager
sudo systemctl restart gremlin-server
sleep 5
# Confirm the agent bound the port
ss -ltnp | grep 9404

If a firewall sits between the node and Prometheus, open the port to the Prometheus host only — never publish 9404 broadly, because the endpoint exposes internal operational detail.

bash 
# Allow the Prometheus server to reach the exporter (example: firewalld)
sudo firewall-cmd --permanent --add-rich-rule=\
'rule family=ipv4 source address=10.0.2.20/32 port port=9404 protocol=tcp accept'
sudo firewall-cmd --reload

Verify: the port is listening and owned by the java process.

bash 
ss -ltnp | grep 9404 | grep -q java && echo "agent listening" || echo "agent NOT bound"

Step 4 — Confirm JanusGraph series on the exporter

Before touching Prometheus, prove the exporter itself serves the renamed metrics. This isolates an exporter problem from a scrape problem — if this step passes and the target is still down later, the fault is network or Prometheus config, not the agent.

bash 
curl -s localhost:9404/metrics | grep janusgraph | head -20

You should see renamed series such as janusgraph_store_operation_latency_seconds{store="edgestore",operation="getSlice",quantile="0.99"} and janusgraph_db_cache_misses_total. If the grep is empty but other metrics exist, the beans are not registered — recheck metrics.enabled and metrics.jmx.enabled.

Verify: at least one storage-latency series with the expected labels is present.

bash 
curl -s localhost:9404/metrics \
  | grep -c 'janusgraph_store_operation_latency_seconds{'

A count of zero means the rename rules did not match the live object names; open jconsole, read one bean’s exact ObjectName, and reconcile the regex.

Step 5 — Add the Prometheus scrape job and query

Add a scrape job targeting every Gremlin Server node on 9404, then reload Prometheus and confirm the target is healthy.

yaml 
# prometheus.yml
scrape_configs:
  - job_name: janusgraph-jmx
    metrics_path: /metrics
    scrape_interval: 15s
    scrape_timeout: 10s
    static_configs:
      - targets: ["gremlin-1:9404", "gremlin-2:9404", "gremlin-3:9404"]
        labels: {tier: storage-backend}
bash 
# Reload without a full restart
curl -s -X POST http://localhost:9090/-/reload

Verify: the target is UP and a PromQL query returns data.

bash 
# Target health
curl -s http://localhost:9090/api/v1/targets \
  | jq '.data.activeTargets[] | select(.labels.job=="janusgraph-jmx") | {instance:.labels.instance, health:.health}'

# p99 storage read latency across the fleet
curl -s http://localhost:9090/api/v1/query \
  --data-urlencode 'query=max by (store) (janusgraph_store_operation_latency_seconds{operation="getSlice", quantile="0.99"})' \
  | jq '.data.result'

A non-empty result with health: "up" confirms the whole pipeline. The healthy ranges and full PromQL catalogue for these series live in the storage and index latency metrics reference.

Fallback and rollback procedures

Each step has a defined recovery path. Validate between actions rather than stacking changes.

If Step 1 fails (jar or flag missing). The grep returns nothing or the jar test prints MISSING. Re-download the agent jar to the referenced path and confirm the JAVA_OPTIONS export runs in the same shell or unit that launches the server — a flag set in a login shell will not reach a systemd service.

If Step 2 fails (YAML will not parse). The agent logs a config error and starts without exporting. Never edit the YAML with a tool that inserts tabs; re-indent with spaces, re-run the yaml.safe_load check, and only then restart.

If Step 3 leaves the port unbound. The java process is up but nothing listens on 9404. The agent almost always failed to load its config — check the Gremlin Server log for a jmx_exporter or MalformedObjectNameException line, fix the referenced cause, and restart. If the port is held by a stale process, ss -ltnp | grep 9404 names the PID to kill.

If Step 4 shows no janusgraph series. Distinguish two cases: no metrics at all means the agent did not bind (return to Step 3); JVM metrics present but no janusgraph means the beans are unregistered (enable metrics.enabled and metrics.jmx.enabled and restart) or the rename regex is wrong (reconcile against the live ObjectName).

If Step 5 shows the target DOWN. The exporter serves locally but Prometheus cannot reach it. Test connectivity from the Prometheus host with curl -s http://gremlin-1:9404/metrics | head; a connection refused points at the firewall rule from Step 3, a timeout at a network path or security group.

To roll the change back entirely. Remove the -javaagent fragment from JAVA_OPTIONS, restart Gremlin Server, and delete the janusgraph-jmx scrape job before reloading Prometheus. The JVM returns to its pre-agent state with no residual listener.