Schema Evolution and CI Gating

Uncoordinated schema mutations are the fastest way to take a production JanusGraph cluster from healthy to full-table-scan in a single merge. The failure is quiet: a property key registered with the wrong data type, a mixed index that never leaves REGISTERED, or an analyzer change applied without a reindex all commit successfully and only surface as query latency days later. This guide — part of the Graph Schema Validation & Modeling Strategies reference — treats the schema as a versioned, stateful artifact and shows how to gate every mutation through CI so a breaking change is blocked before it reaches storage, and a non-breaking one is not merged until its index has actually reached ENABLED. It assumes the type contract established in Vertex and Edge Validation and the index definitions from Property Indexing Rules already exist; CI gating is the machinery that keeps those definitions from drifting as the graph evolves.

The pipeline below shows where CI intercepts a schema change — computing a diff, blocking breaking mutations, and waiting for the index to enable before it allows the merge.

Core Configuration & Consistency Tuning

Schema evolution is only as reliable as the consistency and locking configuration underneath it. JanusGraph serializes every schema mutation through a distributed lock backed by the storage engine, so the lock timeouts, consistency levels, and constraint flags in janusgraph.properties directly determine whether concurrent CI runners deadlock or a half-applied mutation leaves the schema wedged. The following block is a production baseline for a CQL backend (Cassandra or ScyllaDB) with an Elasticsearch-compatible mixed index:

# janusgraph-production.properties
storage.backend=cql
storage.hostname=10.0.1.10,10.0.1.11,10.0.1.12
storage.cql.keyspace=janusgraph_prod
storage.cql.local-datacenter=dc1
storage.cql.read-consistency-level=QUORUM
storage.cql.write-consistency-level=QUORUM

# Fail unregistered / out-of-type writes at ingestion, not at query time
schema.default=none
schema.constraints=true

# Serialize concurrent schema mutations without deadlocking CI runners
storage.lock.wait-time=120000
storage.lock.renew-timeout=60000
storage.lock.clean-expired=true

# Mixed-index backend and creation-time settings (fixed at index creation)
index.search.backend=elasticsearch
index.search.hostname=10.0.2.10,10.0.2.11
index.search.elasticsearch.client-only=true
index.search.elasticsearch.create.ext.number_of_shards=3
index.search.elasticsearch.create.ext.number_of_replicas=1
index.search.elasticsearch.create.ext.refresh_interval=5s

Four operational constraints govern this block:

1. schema.default=none plus schema.constraints=true are non-negotiable for gated deployments. Together they force every property key, vertex label, and edge label to be registered before use and reject out-of-type assignments at write time. This moves validation from the query path to ingestion, which is the entire premise CI gating depends on — the schema becomes a closed contract that a diff can be computed against.
2. Lock timeouts must exceed your longest legitimate schema transaction. The 120s wait and 60s renew accommodate several CI runners contending for the schema lock during a migration window. Set them too low and a slow reindex causes a lock-expiry mid-mutation; too high and a crashed runner holds the lock long enough to stall the pipeline. Keep storage.lock.clean-expired=true so an orphaned lock from a killed job is reclaimed automatically.
3. QUORUM writes are what make a schema mutation durable before acknowledgment. A schema change acknowledged at ONE can be lost to a node failure seconds later, leaving CI convinced the mutation landed. If your topology spans regions, use LOCAL_QUORUM and align it with your Replication Strategies so the schema keyspace replicates the same way the data does.
4. create.ext.* settings only apply at index creation. number_of_shards, number_of_replicas, and refresh_interval are inert once a mixed index exists. CI must assert these values against the staging index before the first mutation, because correcting them later requires a full drop-and-recreate.

For the storage-side details behind the QUORUM mechanics and quorum write-latency model, the Cassandra backend setup guide covers consistency-level selection, and the ScyllaDB migration reference notes where the shard-per-core model changes the lock-contention profile. Authoritative parameter definitions live in the official JanusGraph configuration reference.

Index Synchronization Protocol

The reason CI cannot simply “apply and merge” is that JanusGraph splits a schema mutation across two consistency domains. The storage backend commits type registration and composite indexes synchronously — they are part of the same mutation as the data. Mixed indexes on Elasticsearch or OpenSearch are eventually consistent: JanusGraph commits the graph mutation first, then dispatches the index update asynchronously and drives it through a lifecycle of INSTALLED → REGISTERED → ENABLED. A gate that merges while the index is still REGISTERED ships a change that the query planner cannot use, silently falling back to full cluster scans. The consistency tradeoff this exploits is analyzed in Eventual vs Strong Consistency; the practical rule for CI is that the merge gate must poll the index to a terminal state before it passes.

The states and the actions that drive them:

The lifecycle the gate polls, and the single transition that decides whether a merge is safe:

The polling pattern is ManagementSystem.awaitGraphIndexStatus, which blocks server-side until the target status is reached or a timeout expires. Bound it explicitly — a 300s ceiling is a reasonable default for a staging replica, but size it against the row count being reindexed, not a fixed guess. The lag metric to alert on is the wall-clock time an index spends in REGISTERED: if it plateaus rather than advancing to ENABLED, an index instance is unreachable or a reindex job stalled, and the gate should fail closed rather than time out into an ambiguous state. Where the mixed index spans multiple shards, confirm the alignment described in Mixed Index Routing so a straggler shard does not hold the whole index in REGISTERED.

Python Integration Pattern

The CI job needs a client that applies the schema transactionally, rolls back cleanly on failure, and polls the index to a terminal state — all with the retry discipline a distributed lock demands. Management API calls run server-side, so the class below submits them as Groovy scripts over gremlin-python, wraps each mutation in an explicit openManagement()/commit()/rollback() boundary, and delegates index polling to awaitGraphIndexStatus:

import time
import logging
from gremlin_python.driver.client import Client

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)


class SchemaEvolutionManager:
    def __init__(self, ws_url: str, timeout: int = 300, max_retries: int = 5):
        self.ws_url = ws_url
        self.timeout = timeout
        self.max_retries = max_retries

    def _connect_with_retry(self) -> Client:
        for attempt in range(self.max_retries):
            try:
                client = Client(self.ws_url, "g")
                client.submit("g.V().limit(1)").all().result()
                return client
            except Exception as exc:
                backoff = min(2 ** attempt * 2, 30)
                logger.warning(
                    "Connection failed (attempt %s); retry in %ss: %s",
                    attempt + 1, backoff, exc,
                )
                time.sleep(backoff)
        raise ConnectionError("Gremlin connection failed after max retries.")

    def apply_schema(self, groovy_body: str) -> bool:
        # groovy_body is generated from the versioned schema DSL and must be
        # idempotent: guard every make*() call behind a containsPropertyKey /
        # containsVertexLabel check so a re-run is a no-op, not a duplicate.
        client = self._connect_with_retry()
        script = f"""
            mgmt = graph.openManagement()
            try {{
                {groovy_body}
                mgmt.commit()
                'COMMITTED'
            }} catch (Exception e) {{
                mgmt.rollback()
                throw e
            }}
        """
        try:
            result = client.submit(script).all().result()
            committed = bool(result) and result[0] == "COMMITTED"
            logger.info("Schema transaction committed: %s", committed)
            return committed
        except Exception as exc:
            logger.error("Schema application failed, rolled back: %s", exc)
            raise
        finally:
            client.close()

    def await_index_enabled(self, index_name: str) -> bool:
        client = self._connect_with_retry()
        script = f"""
            org.janusgraph.graphdb.database.management.ManagementSystem
                .awaitGraphIndexStatus(graph, '{index_name}')
                .status(org.janusgraph.core.schema.SchemaStatus.ENABLED)
                .timeout({self.timeout}, java.time.temporal.ChronoUnit.SECONDS)
                .call()
                .getSucceeded()
        """
        try:
            result = client.submit(script).all().result()
            if result and bool(result[0]):
                logger.info("Index %s reached ENABLED.", index_name)
                return True
            raise TimeoutError(
                f"Index {index_name} not ENABLED within {self.timeout}s"
            )
        finally:
            client.close()

Two properties make this safe to run inside a pipeline. First, idempotency: because a gated pipeline may retry a job or replay a batch, groovy_body must guard every makePropertyKey/makeVertexLabel call behind a mgmt.containsPropertyKey(...) check so a second run is a no-op rather than a duplicate-registration error. Second, atomic rollback: any exception inside the transaction triggers mgmt.rollback() before it propagates, so a failed mutation never leaves the schema half-written. When the change adds a mixed index, the job calls apply_schema to register the definition, then await_index_enabled to block the merge until the index is live — the two-phase shape the Index Synchronization Protocol requires. For the query suite that runs against staging afterward, keep field mappings aligned with the analyzer conventions documented alongside the OpenSearch sync patterns and Elasticsearch integration guides so a passing dry-run predicts production behavior.

The end-to-end promotion path the CI job orchestrates:

1. Extract the schema definition from version control (the YAML/JSON DSL, the single source of truth the diff is computed from).
2. Provision an ephemeral staging cluster with production topology, JVM heap ratios, and index shard counts.
3. Compute the diff against the current production schema and classify each change as additive (safe) or breaking (a removed key, a narrowed cardinality, an analyzer change on an existing field).
4. Block promotion on any breaking change and require an explicit migration plan; apply additive changes via apply_schema.
5. Call await_index_enabled for every new mixed index; fail closed on timeout.
6. Run the synthetic query suite and assert index utilization (no full scans) before allowing the merge.

Connection Lifecycle & Pool Management

A CI runner is not a long-lived application server, and treating it like one is a common source of flaky gates. Each pipeline stage should open its Gremlin connection, do its work, and close it in a finally block — exactly as the class above does — rather than holding a pool open across stages where an idle socket can be reaped by a load balancer between the apply and the poll. Size any pool you do keep to the number of concurrent schema operations, which for gating is almost always one: schema mutations serialize on the storage lock regardless of client parallelism, so a pool larger than one buys nothing and multiplies lock contention.

Three rules keep the lifecycle clean:

• Idle timeout below the infrastructure’s socket reaper. If a proxy or the Gremlin Server closes idle connections at 60s, set the client idle timeout under that and validate on borrow (g.V().limit(1)) so a dead socket is detected and replaced, not handed to await_index_enabled mid-poll.
• Retry only the connect, never the mutation blindly. _connect_with_retry retries connection establishment with exponential backoff; a failed mutation must not be retried until the transaction has demonstrably rolled back, or the retry races the original and can double-register.
• Separate the CI service account from application pools. Run gating through its own credentials and connection budget so a stuck pipeline cannot exhaust the driver pool that live traffic depends on. The sizing math and starvation symptoms are covered in Connection Pooling, which is the reference for why an undersized live pool can look identical to schema-induced latency.

Diagnostics & Operational Fallbacks

When a gated mutation misbehaves, the symptom is rarely a clean error — it is a stalled state or a query that has quietly gone slow. The table maps what you observe to the command that confirms the cause and the action that clears it.

When a mutation fails mid-flight, the index is typically left in REGISTERED and the schema lock may still be held. The rollback runbook is: disable the partial index with mgmt.updateIndex(index, SchemaAction.DISABLE_INDEX), clear the lock via mgmt.rollback() and force cluster-wide cache invalidation, then reindex if the mutation altered analyzer configuration. Wire these failure signals into the notification path described in Alert Routing for Violations so a stalled REGISTERED index or a spike in SchemaViolationException pages the on-call engineer rather than waiting to be discovered as query latency.

Standing operational discipline for this seam: keep the schema DSL in version control as the single source of truth, never mutate production schema outside the gated pipeline, and always assert index state before traffic routing shifts to an updated topology.

Frequently Asked Questions

Why gate schema changes in CI instead of applying them manually? Because the failure mode is silent. A mis-typed property key or an index left in REGISTERED commits without error and only surfaces as query latency later. A gate that diffs the schema, blocks breaking changes, and waits for ENABLED turns a class of production incidents into a failed pipeline run.

What counts as a breaking schema change? Removing a property key or label, narrowing a cardinality, changing a data type, or altering the analyzer on an existing mixed-index field. JanusGraph treats a registered key’s type and cardinality as immutable, so these require registering a new key and migrating rather than editing in place — which is exactly why the diff must block them.

Why must the merge wait for the index to reach ENABLED? Composite indexes commit synchronously with the data, but mixed indexes on Elasticsearch or OpenSearch are eventually consistent and pass through INSTALLED → REGISTERED → ENABLED. Merging while the index is still REGISTERED ships a change the query planner cannot use, causing full cluster scans until it happens to enable.

How do I make schema application idempotent for pipeline retries? Guard every makePropertyKey / makeVertexLabel call behind a mgmt.containsPropertyKey(...) check so a replayed job is a no-op instead of a duplicate-registration error, and wrap the whole mutation in openManagement/commit with a rollback on any exception so a failed run never leaves the schema half-written.

An index is stuck in REGISTERED — how do I recover? It means an index instance never acknowledged the definition, usually because a JanusGraph node is unreachable or a reindex stalled. Confirm every instance is up, run REINDEX through the Management API, and if a node is dead remove its stale registration with mgmt.forceCloseInstance(...) before re-polling to ENABLED.

Related

• Up a level: Graph Schema Validation & Modeling Strategies — the parent reference for the type contract this pipeline gates.
• Vertex and Edge Validation — the multiplicity and cardinality rules a schema diff is validated against.
• Property Indexing Rules — the composite and mixed index definitions this gate promotes.
• Alert Routing for Violations — wiring a stalled index or constraint violation to the on-call path.
• Eventual vs Strong Consistency — why the mixed-index async window forces the merge to wait for ENABLED.
• Connection Pooling — sizing the CI service account’s driver connections so gating cannot starve live traffic.