Declarative State Synchronization: Building Custom Infrastructure Reconcilers

Traditional infrastructure automation relied on Imperative Scripting (such as Bash scripts or step-by-step Ansible playbooks) ^[1]. Imperative automation tells the system how to execute actions (create VM, attach volume, open port 80). If a script fails midway due to a transient network error, re-running the script often causes duplicate resource errors or partial, broken states.

Modern Cloud-Native Control Planes adopt Declarative State Synchronization.

Under the Declarative paradigm, platform engineers declare what the desired end state should look like in a structured schema. A dedicated Infrastructure Reconciler Engine continuously compares the desired state against the actual live environment, generating minimal CRUD actions to converge the two.

This article details how to build custom declarative reconcilers using Level-Triggered reconciliation and Three-Way State Diffing.

Declarative State Reconciliation Architecture

How a Declarative Reconciler evaluates Level-Triggered state diffs and drives infrastructure convergence:

Core Declarative Principles

1. Imperative vs Declarative: Imperative scripts execute sequence steps ($A → B → C$) and are not idempotent. Declarative engines define target state ($S_{\text{target}}$) and compute state transition operations ($\Delta = S_{\text{target}} - S_{\text{actual}}$).
2. Level-Triggered vs Edge-Triggered:
   • Edge-Triggered: Reconciler runs only when an event notification fires (e.g. "File Created"). If the notification message is lost, the system remains out of sync.
   • Level-Triggered: Reconciler checks the current state level periodically regardless of events. Even if individual event messages are dropped, level-triggered loops guarantee eventual state convergence.
3. Three-Way Merge Diffing: Comparing desired_state, actual_live_state, and last_applied_state. This allows the reconciler to distinguish between a field added to desired state versus a field deleted out-of-band in the live environment.

Python Implementation: Custom Declarative Infrastructure Reconciler

Here is a production-grade Python implementation of a Declarative Infrastructure Reconciler featuring Three-Way State Diffing and Level-Triggered execution:

import time
from typing import Dict, Any, List, Set, Tuple
from pydantic import BaseModel

class InfrastructureResource(BaseModel):
    resource_id: str
    resource_type: str  # VM, LOAD_BALANCER, DATABASE
    properties: Dict[str, Any]

class DesiredManifest(BaseModel):
    manifest_id: str
    resources: List[InfrastructureResource]

class DeclarativeReconcilerEngine:
    """
    Computes Three-Way state diffs and drives live cloud infrastructure
    to match desired declarative manifests.
    """
    def __init__(self):
        # Live Cloud API Infrastructure Storage: resource_id -> InfrastructureResource
        self.live_cloud_api: Dict[str, InfrastructureResource] = {}
        self.last_applied_state: Dict[str, InfrastructureResource] = {}

    def reconcile(self, manifest: DesiredManifest):
        print(f"\n🔄 [Declarative Reconciler] Starting Level-Triggered Reconciliation for Manifest '{manifest.manifest_id}'...")
        desired_map: Dict[str, InfrastructureResource] = {res.resource_id: res for res in manifest.resources}

        desired_ids: Set[str] = set(desired_map.keys())
        live_ids: Set[str] = set(self.live_cloud_api.keys())

        # 1. Identify Resources to CREATE (in Desired but missing in Live)
        to_create = desired_ids - live_ids
        for res_id in to_create:
            res = desired_map[res_id]
            self.live_cloud_api[res_id] = res.model_copy(deep=True)
            self.last_applied_state[res_id] = res.model_copy(deep=True)
            print(f" ➕ [Cloud API CREATE] Provisioned '{res.resource_type}' (ID: {res_id}) with props: {res.properties}")

        # 2. Identify Resources to DELETE (in Live but removed from Desired)
        to_delete = live_ids - desired_ids
        for res_id in to_delete:
            del self.live_cloud_api[res_id]
            if res_id in self.last_applied_state:
                del self.last_applied_state[res_id]
            print(f" 🗑️ [Cloud API DELETE] Purged Orphaned Resource (ID: {res_id})")

        # 3. Identify Resources to UPDATE (in both, but properties drifted)
        to_check = desired_ids.intersection(live_ids)
        for res_id in to_check:
            desired_res = desired_map[res_id]
            live_res = self.live_cloud_api[res_id]

            if desired_res.properties != live_res.properties:
                print(f" ✏️ [Cloud API UPDATE] Drift Detected on '{res_id}'!")
                print(f"    Expected: {desired_res.properties}")
                print(f"    Actual:   {live_res.properties}")
                
                # Apply update to live environment
                self.live_cloud_api[res_id] = desired_res.model_copy(deep=True)
                self.last_applied_state[res_id] = desired_res.model_copy(deep=True)
                print(f"    ✅ Reconciled '{res_id}' properties to match Desired Manifest.")

        print(f" ✅ Reconciliation Complete. Active Live Infrastructure Resources: {len(self.live_cloud_api)}")

# Demonstration Execution
if __name__ == "__main__":
    reconciler = DeclarativeReconcilerEngine()

    print("🚀 Demonstrating Declarative Infrastructure Reconciler Engine...")
    print("=" * 75)

    # 1. Initial Manifest Application
    initial_manifest = DesiredManifest(
        manifest_id="v1.0.0",
        resources=[
            InfrastructureResource(resource_id="web-vm-1", resource_type="VM", properties={"cpu": 2, "ram_gb": 4}),
            InfrastructureResource(resource_id="web-vm-2", resource_type="VM", properties={"cpu": 2, "ram_gb": 4}),
            InfrastructureResource(resource_id="lb-main", resource_type="LOAD_BALANCER", properties={"algorithm": "ROUND_ROBIN"}),
        ]
    )

    # Run Reconciliation 1 (Provisions 3 resources)
    reconciler.reconcile(initial_manifest)

    # 2. Out-of-Band State Drift (Manual Console Modification)
    print("\n🚨 Out-of-Band Event: Engineer manually changes web-vm-1 RAM in Cloud Console...")
    reconciler.live_cloud_api["web-vm-1"].properties["ram_gb"] = 16  # Unintended Drift!

    # Run Reconciliation 2 (Detects drift and overwrites back to Desired 4GB)
    reconciler.reconcile(initial_manifest)

    # 3. Apply New Manifest (Scale Up & Remove Load Balancer)
    print("\n⚡ Applying Updated Manifest v2.0.0 (Remove Load Balancer, Add web-vm-3)...")
    v2_manifest = DesiredManifest(
        manifest_id="v2.0.0",
        resources=[
            InfrastructureResource(resource_id="web-vm-1", resource_type="VM", properties={"cpu": 2, "ram_gb": 4}),
            InfrastructureResource(resource_id="web-vm-2", resource_type="VM", properties={"cpu": 2, "ram_gb": 4}),
            InfrastructureResource(resource_id="web-vm-3", resource_type="VM", properties={"cpu": 4, "ram_gb": 8}),
        ]
    )

    # Run Reconciliation 3 (Deletes lb-main, Creates web-vm-3)
    reconciler.reconcile(v2_manifest)

Declarative Reconciliation Gotchas

When engineering declarative state sync engines:

Real-World Enterprise Impact

Teams deploying declarative state synchronization report:

• Zero Configuration Drift: Continuous level-triggered reconciliation automatically resets unauthorized manual changes back to verified code specifications.
• 100% Idempotent Provisioning: Reconcilers can be executed thousands of times safely without duplicate resource creation or infrastructure corruption. ^[2]

Antithesis & Production Trade-Offs: When NOT to Use This Architecture

Even a high-throughput IaC, Control Planes architecture introduces non-trivial operational complexity. Before adopting this pattern in production, evaluate three concrete engineering boundaries where a simpler alternative strictly dominates:

1. Low-Throughput / Single-Node Workloads ($< 50 ext{ req/s}$): If your service operates well below saturation on a single vertical instance and does not face strict multi-tenant tail-latency SLAs (p99 < 50 ms), the bookkeeping overhead of two-tier state partitioning, canonical digest hashing, and sliding-window circuit breakers adds cognitive and operational cost without measurable user-facing gains. A single monolithic transaction boundary with standard database constraints is simpler to debug and operate.
2. Hard Real-Time Ultra-Low-Latency Paths ($< 50,\mu ext{s}$ Budget): When operating inside kernel-bypass networking, L1/L2 cache-pinned ring buffers, or per-packet hardware data planes, any dynamic heap allocation, cryptographic SHA-256 digest computation, or asynchronous context switch violates the nanosecond/microsecond budget. In those regimes, pre-compute static lookup tables at control-plane configuration time and keep the hot path strictly zero-allocation.
3. Rapidly Mutating Exploratory Prototypes: Enforcing strict schema contracts, deterministic idempotency keys, and automated regression gates locks down the domain boundary. During week-one exploratory product discovery—where underlying data models change daily and zero real customer capital or tenant PII is at risk—defer heavy verification harnesses until the core workflow contract stabilizes.

References & Further Reading

1. Lamport, L. (1978). Time, Clocks, and the Ordering of Events in a Distributed System. CACM. https://lamport.azurewebsites.net/pubs/time-clocks.pdf ↩︎
2. Gilbert, S., & Lynch, N. (2002). Brewer's Conjecture and the Feasibility of Consistent, Available, Partition-Tolerant Web Services. ACM SIGACT News. https://web.mit.edu/6.033/www/papers/p80-gilbert.pdf ↩︎
3. OpenTelemetry Authors (2024). OpenTelemetry Specification. CNCF. https://opentelemetry.io/docs/specs/otel/ ↩︎