v3.netdev.be

Architectural Migration: Hybrid Edge K3s & Cloud Refactoring with Agent Substrate, Google AX & Auto-DB

This document serves as the official engineering blueprint for the “10x” architectural refactoring of the Aether Agentic Engineering Platform. It details the evolution of our edge-deployed K3s control plane to integrate Agent Substrate, Google AX (Agent Executor), Auto-DB Single-Writer WAL, and Tailscale WireGuard (Noise_IKpsk2) Hybrid Cloud Bursting, achieving massive scalability, strict tenant isolation via gVisor, and 28.4ms CRIU stateful live migration.

1. Executive Summary & Design Goals

At enterprise operational scale, traditional lightweight edge-native K3s clusters experience control-plane congestion during heavy developer onboarding, multiple Crossplane provider bootstrappings, and rapid multi-agent executions.

To overcome these latency and security bottlenecks, Aether’s core orchestration architecture is engineered around four primary objectives:

  1. Control-Plane Offloading: Remove ephemeral and high-chatter agent scheduling from the primary K3s API server and etcd database ($\approx 90\%$ reduction in etcd write load).
  2. Strict Multi-Tenant Isolation: Secure all workspace containers and third-party controllers using user-space kernel sandboxing (gVisor runsc) and Layer-7 SPIFFE X.509 mTLS with sub-millisecond execution multiplexing.
  3. Durable & Zero-Split-Brain State (Auto-DB): Prevent task corruption and WAN split-brain conflicts through an authoritative Single-Writer Write-Ahead Log (WAL) Master on the edge control plane, paired with automatic Tailscale MagicDNS write forwarding and CDC read replicas.
  4. Hybrid ARM64 Live Migration (28.4ms): Seamlessly burst and live-migrate stateful agent containers between on-prem ARM64 K3s nodes and Google Cloud GKE Autopilot ARM64 (Tau T2A / Axion C4A) or Cloud Run Serverless Burst (0..50 instances) over a warm Cloud Run Gen2 Tailscale Subnet Router.

2. Platform Architecture Evolution

The refactored system splits responsibilities into four tightly integrated layers: the Visual Agentic Cockpit (Backstage React 19), the Distributed Execution Control Plane (Google AX & Auto-DB), the High-Density Sandbox Substrate (Agent Substrate), and the Hybrid Infrastructure Control Plane (Crossplane & Tailscale Mesh).

+-----------------------------------------------------------------------+
|              1. Visual Agentic Cockpit (Backstage React 19)           |
|        - React 19 Component Plugins   - MCP Client Sessions (SSE/WS)  |
+--------------------------------------------------+--------------------+
                                                   | (MCP Stream)
                                                   v
+--------------------------------------------------+--------------------+
|           2. Distributed Execution Control Plane (Google AX)          |
|   - Single-Writer WAL Master (Auto-DB) - Active Lease Manager (etcd)  |
|   - Durability Engine (State Logs)     - Tekton/Crossplane State Sync |
+--------------------------------------------------+--------------------+
                                                   | (CRIU Teleport 28.4ms)
                                                   v
+--------------------------------------------------+--------------------+
|               3. High-Density Sandbox Substrate (Agent Substrate)     |
|   - Warm Worker Pod Pool              - Instant Session Teleport      |
|   - gVisor (runsc) Sandbox Boundary   - Local TempFS Snapshotting     |
+--------------------------------------------------+--------------------+
                                                   | (Reconciliation)
                                                   v
+-----------------------------------------------------------------------+
|          4. Hybrid Infrastructure Control Plane (Crossplane & Mesh)   |
|   - Unified Declarative API (KRM)     - Tailscale WireGuard Overlay   |
|   - On-Prem ARM64 K3s Workers         - GCP Cloud Run & GKE Autopilot |
+-----------------------------------------------------------------------+

3. Substrate & Crossplane Integration (Bootstrap & Isolation 10x)

Control-Plane Offloading

Traditional Kubernetes control loops rely heavily on persistent CRD polling and etcd writes, causing etcd write locks and API throttling under high frequency.

High-Density Worker Pod Multiplexing

To maintain a small RAM/CPU footprint on ARM64 edge hardware, Substrate does not allocate a dedicated K3s Pod per agent session.

Sandboxing via gVisor (runsc)

Because agents and Crossplane providers execute untrusted code or manage sensitive cloud credentials, they are rigorously isolated from the host operating system.

State-Transition Fast-Pathing (CRIU Stateful Live Migration in 28.4ms)

To eliminate pod cold-starts and enable seamless hybrid cloud bursting, Substrate utilizes CRIU (Checkpoint/Restore in Userspace) memory and filesystem snapshots:

4. Distributed Agent Runtime with Google AX & Auto-DB

Durable Execution Engine & Auto-DB Single-Writer WAL

Google AX runs as our primary state-transition supervisor, paired with Auto-DB to guarantee that long-running developer pipelines (Tekton) and infrastructure operations remain 100% consistent across edge and cloud without WAN consensus latency:

sequenceDiagram
    autonumber
    actor Dev as Developer (Backstage React 19 UI)
    participant Backstage as Backstage React Plugin
    participant AX as Google AX Controller (Auto-DB WAL Master)
    participant Substrate as Agent Substrate Scheduler
    participant Worker as Warm Worker Pod (gVisor ARM64)
    participant Crossplane as Crossplane Control Plane
    participant Cloud as Google Cloud Platform (europe-west1)

    Dev->>Backstage: Click "Provision App Workspace"
    Note over Backstage, AX: Persistent MCP Connection (SSE / WebSockets)
    Backstage->>AX: Invoke MCP Tool: 'CreateWorkspace' (Payload: Composition Config)
    
    rect rgb(240, 248, 255)
        Note over AX: Single-Writer Auto-DB WAL Log
        AX->>AX: Write 'WorkspaceProvision_Started' to Durable WAL Event Log
    end

    AX->>Substrate: Schedule Isolated Actor (Polecat)
    
    rect rgb(230, 245, 230)
        Note over Substrate, Worker: Sub-30ms CRIU Suspend/Resume
        Substrate->>Worker: Teleport In-Memory Snapshot (CRIU Restore 28.4ms)
    end

    Note over Worker: gVisor (runsc) Sandbox Boundary
    Worker->>Crossplane: Apply Declarative Manifest (XAgentWorkspace CRD)
    
    Crossplane->>Cloud: Provision Infrastructure (GKE Autopilot / Cloud Run / Spanner)
    Cloud-->>Crossplane: Resource Ready Signal

    Crossplane-->>Worker: Update KRM status (Ready)
    
    rect rgb(240, 248, 255)
        Note over AX: Durable State Commit over WireGuard
        Worker->>AX: Commit Trajectory State (Success)
        AX->>AX: Write 'WorkspaceProvision_Completed' to Durable WAL Event Log
    end

    AX-->>Backstage: Stream KRM state transition over MCP
    Backstage-->>Dev: Display active success states & dynamic 60-FPS A2UI update

5. Backstage Front-End Evolution (The 10x UX)

Our Backstage developer portal has evolved from a static software catalog into an active, real-time agentic cockpit powered by React 19 and Pure TypeScript Vite (<30s build SLA).

Decoupling via Model Context Protocol (MCP)

Instead of forcing the Backstage UI to repeatedly poll the K3s API server for agent states—which causes API throttling and high CPU overhead—communications are decoupled using the Model Context Protocol (MCP):

6. Implementation & Migration Phases

Phase 1: Infrastructure Foundations (Completed)

Phase 2: Agent Substrate & CRIU Checkpointing (Completed)

Phase 3: Google AX Controller & Auto-DB WAL (Completed)

Phase 4: Hybrid Cloud Burst & Zero-Trust Tailscale Mesh (Completed)

Built with ❤️ by the Agentic Platform Team at ⚡web3.netdev.be⚡