Troubleshooting Elasticsearch Index Lag

When the interval between a JanusGraph storage commit and the moment that vertex becomes searchable keeps widening, you are watching index lag accumulate, and this guide is the ordered procedure for measuring it, tracing it to a single cause, and driving the stale-read window back under your SLA. It sits under the Elasticsearch Integration reference and narrows that seam to one recurring incident: the dispatch pipeline that was keeping up yesterday is now falling behind, has() predicates return documents seconds late, and nobody changed the schema. The failure this prevents is the reflexive fix — someone lowers refresh_interval or drops bulk-size on a hunch, thrashes segment merges, and doubles the lag they were trying to remove. Every step below is a measurement you can act on, not a guess, and each ends with a verification you run before touching the next lever.

The three measurement taps along the JanusGraph-to-Elasticsearch dispatch path and the cause each isolates A committed graph mutation flows left to right through the IndexProvider async queue, the Elasticsearch bulk write thread pool, and the per-index refresh cycle before it becomes searchable. Three measurement taps sit on that path: the IndexProvider JMX queue depth reveals producer backpressure, the _cat/thread_pool/write queue and rejected counters reveal a saturated write pool, and the _stats/refresh and merges endpoints reveal refresh thrash or merge throttling. Reading the tap that is rising tells you which single cause is stretching the stale-read window between commit and searchable. stale-read window — committed, not yet searchable Storage commit vertex / edge IndexProvider async queue Bulk write pool _bulk → shards Refresh + merge searchable segment TAP 1 · queue depth IndexProvider JMX bean rising → producer backpressure TAP 2 · write pool _cat/thread_pool/write rejected > 0 → saturation TAP 3 · refresh / merge _stats/refresh · _stats/merge throttled → merge / refresh thrash t₀ commit clock starts read the tap that is rising — it names the one cause to fix before touching any other lever
Three taps sit on the dispatch path. Whichever counter is climbing — queue depth, write-pool rejections, or merge-throttle time — isolates the single cause stretching the stale-read window.

Prerequisites

Confirm each item before you change a property, because the most common wasted incident is tuning the bulk pipeline when the real lag is a saturated Elasticsearch node.

  • JanusGraph 0.6.x or 1.0.x with index.search.backend=elasticsearch and the REST client (index.search.elasticsearch.client-only=true) already wired per the parent reference.
  • Elasticsearch 7.x or 8.x reachable, with cluster health green or yellow — a red index has an unassigned shard and that is a recovery problem, not a lag problem.
  • JMX or a Prometheus scrape enabled on JanusGraph (metrics.enabled=true, metrics.jmx.enabled=true) so the IndexProvider queue is a number, not a hunch.
  • curl and jq on the on-call host, plus read access to the Elasticsearch _cat, _stats, and _nodes/stats endpoints (a monitoring role suffices).
  • A known baseline. You need the lag value from a healthy window to know how far you have drifted — if you have never captured one, Step 1 also establishes it.
  • Write access to the live index _settings through the Elasticsearch API, since create.ext.* properties in janusgraph.properties are inert after the mixed index exists.

Step 1 — Measure the lag at all three taps

Sample the three signals together in one pass so you compare them at the same instant. A single tap in isolation lies: a full queue with an idle write pool is a different fault than a full queue with a rejecting write pool.

bash 
# Tap 1 — IndexProvider async queue depth (JanusGraph side, via Prometheus scrape of JMX)
curl -s http://localhost:9090/api/v1/query \
  --data-urlencode 'query=janusgraph_index_provider_queue_size' | jq '.data.result[].value[1]'

# Tap 2 — Elasticsearch bulk write pool: active, queue, and cumulative rejections
curl -s 'http://es-cluster-01:9200/_cat/thread_pool/write?v&h=node_name,active,queue,rejected,completed'

# Tap 3 — refresh and merge pressure on the mixed index
curl -s 'http://es-cluster-01:9200/janusgraph_mixed/_stats/refresh,merge' \
  | jq '.indices.janusgraph_mixed.total | {refresh_total: .refresh.total,
        refresh_ms: .refresh.total_time_in_millis,
        merge_throttle_ms: .merge.total_throttled_time_in_millis}'

Record all three, then compute the observable lag directly as the delta between a storage commit timestamp and the moment that document answers a mixed-index has() predicate. That end-to-end number is the one your users feel; the three taps only explain it.

Verify: you have four numbers written down — queue depth, write-pool rejected, merge-throttle milliseconds, and the measured commit-to-searchable delta — captured within the same ten-second window.

bash 
# Sanity check that the scrape and ES agree the pipeline is moving at all
curl -s 'http://es-cluster-01:9200/janusgraph_mixed/_stats/indexing?filter_path=**.index_total,**.index_current' | jq .

Step 2 — Localize the cause to one tap

Now branch on which counter is rising, because the fix for each is different and applying the wrong one makes lag worse. Read the taps in this order and stop at the first that is abnormal.

bash 
# Take two samples one refresh_interval apart and diff them
for label in before after; do
  echo "== $label =="
  curl -s 'http://es-cluster-01:9200/_cat/thread_pool/write?v&h=queue,rejected'
  curl -s 'http://es-cluster-01:9200/janusgraph_mixed/_stats/merge' \
    | jq '.indices.janusgraph_mixed.total.merge.total_throttled_time_in_millis'
  [ "$label" = before ] && sleep "${REFRESH_INTERVAL:-1}"
done

Interpret the diff against a single decision:

  • Tap 1 rising, Tap 2 idle — the IndexProvider queue climbs while the write pool has spare capacity. The producer is outrunning the async dispatch, not the search cluster. This is client-side backpressure; the fix is throughput shaping, not Elasticsearch tuning.
  • Tap 2 rejected climbing — the Elasticsearch write thread pool queue overflowed and shed bulk requests. The search cluster cannot absorb the write rate. This is genuine cluster saturation.
  • Tap 3 merge-throttle time climbing fast — segment merges are throttled and refresh is thrashing, almost always because refresh_interval was set too low and each tiny refresh spawns a segment that must then be merged. Lowering refresh to chase freshness caused the lag.
  • Tap 3 refresh count flat but merge time high, nodes hot — the nodes are I/O- or CPU-saturated on merge work; nodetool-equivalent node stats show high merges.current and disk at ceiling. This is node saturation and needs capacity, not config.

Verify: you can name exactly one tap as the cause and state it in a sentence — for example, “Tap 2 rejected rose from 0 to 4,120 in one interval; the write pool is saturated.” If two taps rise together, the upstream one (lower number) is primary.

Step 3 — Apply the fix for that one cause

Change one lever, matched to the tap from Step 2. Live index settings go through the Elasticsearch _settings API because the create.ext.* properties no longer apply once the index exists.

bash 
# CAUSE = Tap 3 refresh thrash → raise refresh_interval to stop segment churn
curl -s -XPUT 'http://es-cluster-01:9200/janusgraph_mixed/_settings' \
  -H 'Content-Type: application/json' -d '{"index": {"refresh_interval": "5s"}}'

# CAUSE = Tap 3 merge saturation on SSD → give the merge scheduler more threads + buffer
curl -s -XPUT 'http://es-cluster-01:9200/janusgraph_mixed/_settings' \
  -H 'Content-Type: application/json' \
  -d '{"index": {"merge.scheduler.max_thread_count": 4}}'

For a producer-side or write-pool cause, the lever lives in janusgraph.properties and takes effect on a Gremlin Server restart:

properties 
# CAUSE = Tap 1 producer backpressure → smaller batches commit-order faster, less queue burst
index.search.elasticsearch.bulk-size=500

# CAUSE = Tap 2 write-pool saturation → cap concurrent bulk sockets so a burst cannot flood the pool
index.search.elasticsearch.http.max-connections=40
index.search.elasticsearch.http.max-connections-per-route=15
# Keep the retry budget below the upstream transaction timeout so a saturated
# cluster surfaces as a clean failure the pipeline can reconcile, not a stall.
index.search.elasticsearch.max-retry-time=180000

Do not lower bulk-size for a merge-thrash cause or raise refresh_interval for a producer-backpressure cause — the levers are not interchangeable, and a hot shard from skewed graph partitions can masquerade as any of these. If one shard is doing all the work, the real fix is shard alignment, covered under Mixed Index Routing.

Verify: the setting is live and the previously rising tap has stopped climbing.

bash 
curl -s 'http://es-cluster-01:9200/janusgraph_mixed/_settings?filter_path=**.refresh_interval,**.merge' | jq .

Step 4 — Verify the stale-read window shrinks

A setting that applied is not a lag that closed. Re-run the Step 1 measurement and confirm the end-to-end commit-to-searchable delta is trending back toward baseline, not just that a counter moved.

python 
import time
from datetime import datetime, timezone
from gremlin_python.driver.driver_remote_connection import DriverRemoteConnection
from gremlin_python.process.anonymous_traversal import traversal

def measure_visibility_lag(ws_endpoint, probe_prop="lag_probe"):
    conn = DriverRemoteConnection(ws_endpoint, "g")
    g = traversal().withRemote(conn)
    token = f"probe-{datetime.now(timezone.utc).timestamp()}"
    t0 = time.monotonic()
    g.addV("lag_probe").property(probe_prop, token).iterate()   # storage commit
    # Poll the MIXED index (has() predicate), not a storage-backed id lookup.
    while time.monotonic() - t0 < 30:
        if g.V().has("lag_probe", probe_prop, token).hasNext():
            lag = time.monotonic() - t0
            print(f"visible after {lag:.2f}s")
            conn.close()
            return lag
        time.sleep(0.25)
    conn.close()
    raise TimeoutError("probe never became searchable within 30s")

# measure_visibility_lag("ws://gremlin-01:8182/gremlin")

Run the probe several times and take the p95, not a single sample — one lucky fast read proves nothing. The window has closed when p95 sits back inside your read-your-writes budget and the tap you fixed stays flat under production load.

Verify: three consecutive probe p95 values are within budget and Tap 1/2/3 are all flat.

bash 
watch -n 10 "curl -s 'http://es-cluster-01:9200/_cat/thread_pool/write?v&h=queue,rejected'"

Fallback and rollback procedures

Change one lever, validate, and only then consider the next. Stacking changes hides which one worked.

If Step 1 cannot read the IndexProvider queue. The JMX bean is absent because metrics.jmx.enabled=true was never set or the scrape target is wrong. Fall back to the Elasticsearch-only taps (_cat/thread_pool/write and _stats/refresh) to localize, and fix the metrics wiring before the next incident — flying blind on Tap 1 means you cannot distinguish producer backpressure from cluster saturation.

If Step 3 raised refresh_interval and lag did not move. The cause was not refresh thrash. Revert to the prior interval so you do not silently widen every read’s window, and re-read Step 2 — the primary tap was almost certainly Tap 2 (write-pool saturation) or node saturation, which refresh_interval cannot touch.

bash 
# Roll refresh_interval back to the value that was live before the change
curl -s -XPUT 'http://es-cluster-01:9200/janusgraph_mixed/_settings' \
  -H 'Content-Type: application/json' -d '{"index": {"refresh_interval": "1s"}}'

If lowering bulk-size increased the queue. Smaller batches multiplied request overhead against a healthy cluster. Restore the prior bulk-size, restart the canary node only, and shift the fix upstream to producer throttling instead of batch shrinking.

If the lag persists past the transaction-log horizon. JanusGraph replays queued index mutations from its transaction log on reconnect, but once drift outlives the retained log the index is missing writes no lever will backfill. At that point stop tuning and run a controlled reindex — the zero-downtime procedure and node-failure recovery paths are in Reindexing & Recovery. Reindex during a maintenance window with bulk-refresh=false so the repair pass does not itself saturate the write pool you just relieved.

Frequently Asked Questions

Why did lowering refresh_interval make my index lag worse instead of better? A lower refresh interval forces Elasticsearch to cut a new segment more often, and each tiny segment must then be merged. Under sustained graph ingestion that multiplies merge I/O until the merge scheduler throttles, which stalls the very writes you wanted to make visible faster. Freshness on routed reads comes from bulk-refresh=wait_for scoped to the specific writes that need it, not from globally shrinking refresh_interval.

How do I tell producer backpressure apart from Elasticsearch saturation? Read two taps at the same instant. If the JanusGraph IndexProvider queue is climbing while _cat/thread_pool/write shows a near-empty queue and zero rejections, the producer is outrunning the dispatch and the search cluster is fine — throttle the writer. If the write pool queue is full and rejected is climbing, the search cluster itself cannot keep up and needs smaller batches, more write threads, or more nodes.

Can I change refresh_interval without rebuilding the mixed index? Yes, but not through janusgraph.properties. The create.ext.refresh_interval key only applies at index creation and is inert afterward. Change the live value with a PUT /<index>/_settings call against the Elasticsearch API; it takes effect immediately without a reindex.

The queue keeps growing even after I fixed the cause — what now? If drift has outlived the JanusGraph transaction-log retention, the index is missing mutations that no live lever will backfill, because replay only covers what is still in the log. Run a controlled reindex from storage during a maintenance window with bulk-refresh=false; the full zero-downtime and recovery procedures live in the Reindexing & Recovery reference.