Configuring Mixed Index Fallback Chains

This guide walks through wiring a JanusGraph mixed-index fallback chain — a primary search backend, a secondary backend, and a bounded composite-index path — so that a degraded Elasticsearch/OpenSearch cluster degrades query latency instead of hard-failing traversals and stalling downstream pipelines. It is the hands-on procedure behind the routing theory in the parent Mixed Index Routing guide: production deployments routinely hit transient index degradation from network partitions, JVM garbage-collection pauses, or shard rebalancing, and a query that raises IllegalArgumentException or times out during that window violates SLOs and triggers backpressure. Configured correctly, the chain shifts traffic to a healthy backend or a composite-index fallback and reverts automatically once the primary recovers.

The state machine below captures the circuit-breaker lifecycle the fallback router moves through.

Circuit-breaker lifecycle of the mixed-index fallback router The router starts in the Primary state, routing traversals to the primary mixed index with query.force-index true. When a health check fails it transitions to the Fallback state, sets force-index false, and serves bounded queries from the composite index — a self-loop it holds for the duration of the outage. Once the primary's replicas report UN and the async queue has drained, it moves to a Recovery state that probes primary health, and when the index status returns to ENABLED it transitions back to Primary and restores force-index true. Primary primary mixed index Fallback composite fallback path Recovery probing primary health health check fails replicas UN · queue drained serve via composite index index status ENABLED

JanusGraph does not natively auto-chain mixed indexes — it dispatches a routed predicate to a single IndexProvider and fails the query if that provider is unreachable. The chain therefore lives at your query proxy or pipeline layer, not inside the graph engine.

Prerequisites

Confirm every item before you start — a fallback chain built on an unverified composite index fails open into full scans, which is worse than the outage it was meant to survive.

  • JanusGraph 0.6+ running against a Cassandra, ScyllaDB, or HBase storage backend, with gremlin-python 3.5+ on the client.
  • Two reachable search backends — a primary and a secondary Elasticsearch 7.x/8.x or OpenSearch 1.x/2.x cluster. If you run mixed engines, reconcile analyzer and mapping differences first per Elasticsearch Integration and OpenSearch Sync Patterns, because a predicate that routes cleanly on one engine can match nothing on the other.
  • A composite index that fully covers the fallback predicates. Every property key the fallback path queries must be bound to an ENABLED composite index. Verify eligibility against Property Indexing Rules — only equality and range predicates on covered keys survive the degraded path.
  • Management-API access (graph.openManagement()) and permission to edit janusgraph.properties and restart the server pool.
  • A settled consistency target. Decide where the fallback path sits on the eventual versus strong consistency spectrum before you cut traffic over — serving from a lagging secondary is a deliberate consistency trade, not an accident.

Step-by-Step Procedure

Step 1 — Declare primary and fallback indexes

Define both mixed indexes explicitly in janusgraph.properties, with strict timeout boundaries so a slow backend cannot exhaust the client thread pool during a state transition.

properties 
# Primary Mixed Index Configuration
index.primary.backend=elasticsearch
index.primary.hostname=es-primary-01,es-primary-02,es-primary-03
index.primary.elasticsearch.client-only=true
index.primary.elasticsearch.http.connection-timeout=2000
index.primary.elasticsearch.http.socket-timeout=5000

# Fallback Mixed Index Configuration
index.fallback.backend=elasticsearch
index.fallback.hostname=os-fallback-01,os-fallback-02
index.fallback.elasticsearch.client-only=true
index.fallback.elasticsearch.http.connection-timeout=1500
index.fallback.elasticsearch.http.socket-timeout=3000

# Query Safety Boundaries
# query.force-index=true forces all traversals through an index; set false
# only after confirming composite index coverage for the fallback path.
query.force-index=true
query.fast-property=true
storage.read-time=10000
storage.write-time=15000

Keep query.force-index=true as the steady-state default. It guarantees that no traversal silently degrades into a full scan while both search backends are healthy; the router toggles it off deliberately only when it drops to the composite path.

Step 2 — Build the circuit-breaker router

Implement the chain in your ingestion or query pipeline as a stateful circuit breaker. This router polls index health on a cooldown, toggles query.force-index at runtime, and executes each attempt against a freshly built traversal.

python 
import time
import requests
from typing import List, Any, Callable
from gremlin_python.process.graph_traversal import __
from gremlin_python.process.traversal import Traversal

class JanusFallbackRouter:
    def __init__(self, primary_es_url: str, graph: Any):
        self.primary_es_url = primary_es_url
        self.graph = graph
        self.state = "primary"
        self.last_health_check = 0.0
        self.cooldown = 5.0  # seconds

    def check_index_health(self) -> bool:
        if time.time() - self.last_health_check < self.cooldown:
            return self.state == "primary"
        try:
            # Validate Elasticsearch/OpenSearch cluster health
            resp = requests.get(f"{self.primary_es_url}/_cluster/health", timeout=2)
            healthy = resp.json().get("status") in ("green", "yellow")
            self.last_health_check = time.time()
            return healthy
        except requests.RequestException:
            self.last_health_check = time.time()
            return False

    def toggle_force_index(self, force: bool) -> None:
        # Runtime configuration update via JanusGraph ManagementSystem.
        # Note: not all runtime properties can be changed this way; verify
        # against JanusGraph's hot-reload documentation before use.
        mgmt = self.graph.openManagement()
        mgmt.set("query.force-index", str(force).lower())
        mgmt.commit()

    def execute_query(self, build_traversal: Callable[[], Traversal], timeout_ms: int = 5000) -> List[Any]:
        # Accept a factory: a traversal cannot be re-iterated once executed, so
        # each attempt must build a fresh one. Per-query timeout is set with the
        # 'evaluationTimeout' option.
        if self.state == "primary" and not self.check_index_health():
            self.state = "fallback"
            self.toggle_force_index(False)

        if self.state == "fallback":
            try:
                # Execute against composite/bounded path
                return build_traversal().with_("evaluationTimeout", timeout_ms).toList()
            except Exception as e:
                raise RuntimeError("Fallback traversal failed. Verify composite index coverage.") from e

        try:
            return build_traversal().with_("evaluationTimeout", timeout_ms).toList()
        except Exception:
            self.state = "fallback"
            self.toggle_force_index(False)
            return build_traversal().with_("evaluationTimeout", timeout_ms).toList()

Step 3 — Define the routing hierarchy

Wire the router to walk the chain in strict order so every degradation has a bounded next hop:

  1. Primary mixed index — the high-throughput search cluster handling all standard predicate queries.
  2. Secondary mixed index — a geographically isolated or lower-tier cluster that absorbs overflow when primary latency crosses the SLO threshold.
  3. Composite index + bounded storage scan — the last resort, restricted to exact-match or range predicates the composite index covers, entered only after query.force-index is toggled to false.

The router must evaluate index health, replication lag, and query complexity before dispatch. Full-text, geoWithin, and open-ended range predicates cannot execute on the composite path — the fallback layer must reject or reshape them rather than let them fall through to an unbounded scan.

Step 4 — Apply and reload

Push the property changes and confirm both index registrations are live:

bash 
# Register/confirm index definitions from the Gremlin console
gremlin> mgmt = graph.openManagement()
gremlin> mgmt.printIndexes()
gremlin> mgmt.commit()

Verification Commands

Validate the chain in a staging environment before promoting it — never trust a fallback path you have not tripped on purpose.

  1. Baseline health. Confirm the primary reports a healthy status and both indexes are registered.

    bash 
    curl -s "http://es-primary-01:9200/_cluster/health?pretty" | grep '"status"'
# expect "green" (or "yellow" on a single-replica staging cluster)

    In the Gremlin console, mgmt.printIndexes() must list both the primary mixed index and the composite fallback index at status ENABLED.

  2. Simulate primary degradation. Throttle or blackhole traffic to the primary hosts with tc qdisc (Linux Traffic Control), or inject a 503 at the API gateway.

  3. Observe the routing shift. Watch the pipeline logs for the primary → fallback transition and confirm query.force-index flips to false within one cooldown window (2 seconds by default).

  4. Confirm composite coverage. Run a bounded, indexed predicate and assert it returns without a full-scan warning:

    python 
    results = router.execute_query(
    lambda: g.V().has("user_id", "u-4417"), timeout_ms=3000
)
assert results, "fallback returned empty — composite index does not cover user_id"

  5. Recovery verification. Restore primary connectivity. After the cooldown expires the router must revert to state == "primary" and toggle query.force-index back to true. Confirm the next query dispatches to the primary index (query latency drops back to the sub-millisecond baseline).

Fallback Procedures

Each step has a distinct failure mode. Recover with the matching rollback rather than retrying blindly.

  • Step 1 fails — server rejects the properties or a backend is unreachable at boot. Revert to a single-index janusgraph.properties, restart the pool, and restore service on the primary alone. Re-add the fallback block only after the secondary cluster answers /_cluster/health. Rollback: git checkout janusgraph.properties && systemctl restart janusgraph.

  • Step 2/3 fails — the router flips to fallback but composite queries raise IllegalArgumentException. This means query.force-index was toggled while the fallback predicates are not composite-covered. Immediately force the router back to primary state, set query.force-index=false globally as a stopgap to accept bounded scans, and cap blast radius with query.page-size to prevent OOM. Then bind the missing composite index before re-arming the chain.

  • Step 4 fails — printIndexes() shows the fallback index stuck below ENABLED. Do not route traffic to it. Complete the index lifecycle (REGISTER_INDEX → ENABLE_INDEX, reindex if the key predated the index) per the JanusGraph index-lifecycle documentation and re-run verification.

  • Fallback returns empty result sets despite valid data. The index trails the storage commit or a mapping drifted. Align the backend commit/refresh interval with the fallback timeout window, confirm the composite index includes every predicate property, and reconcile drift via the sync procedures in OpenSearch Sync Patterns.

  • Router flaps between primary and fallback. Health checks are firing inside GC pauses. Widen cooldown and tune the poll interval against measured GC pause durations; see the Elasticsearch Cluster Health API documentation for the fields to threshold on. Full-stop rollback for any unrecoverable state: restore query.force-index=true, pin the router to primary, and take the degraded backend out of rotation until it is repaired.

Frequently Asked Questions

Why not let JanusGraph fail over on its own? It has no chaining primitive — a routed predicate targets one IndexProvider, and an unreachable provider raises an exception. The chain must live in the proxy or pipeline layer.

Is it safe to leave query.force-index=false permanently? No. With it off, any unindexed predicate silently becomes a full scan. Keep it true in steady state and let the router toggle it only for the duration of a degradation.

What predicates can the composite fallback actually serve? Only equality and bounded range predicates on keys with an ENABLED composite index. Full-text, geo, and open-ended range predicates have no composite equivalent and must be rejected on the fallback path.