Routing Rules That Don't Break at 3am: Designing Dynamic LLM Fallback for High-Availability Agentic Pipelines

1. Policy-Driven Model Routing: Externalizing Fallback Rules from Pipeline Code

If your fallback logic lives inside the pipeline code, data science teams can't adjust it without involving platform engineering — externalize routing rules into a versioned policy layer that the pipeline reads at runtime.

Routing rules as data, not code: store fallback chains, quality thresholds, and latency budgets in a config layer (database row, feature flag, or object store object) the pipeline evaluates per-request, not per-deploy.
Hot-reload without restarts means your async Node.js processor can pull the current routing policy on a TTL-based cache refresh, giving data science teams sub-minute propagation without touching infrastructure.
Version the policy, not the pipeline: treat each routing config as a named artifact with a timestamp and author, so you have a full audit trail when a fallback decision causes a downstream anomaly.

TECHNICAL DEEP DIVE & CODE ARTIFACTS Expand [+]

The framing from CrewAI's 'stop giving your agents database credentials' talk is useful here beyond the security angle — the same principle of abstracting access applies to model routing. An agent (or pipeline step) shouldn't know the endpoint it's calling; it should invoke a routing policy that resolves to an endpoint given current conditions. That indirection is what lets you swap models, adjust fallback order, or raise quality thresholds without touching the pipeline binary. Your routing policy becomes a first-class artifact with its own lifecycle: it gets created, versioned, promoted to production, and rolled back, just like application code.

AIThe governance pattern S&P Global applied to their semantic layer — curating focused domain agents whose composition is managed by a proxy layer rather than hardcoded in each consumer — is structurally identical to what good model routing needs. Each LLM in your fallback chain is a domain-scoped capability; the routing layer composes them according to policy. This separation means your data science team owns the policy (what model handles what conditions at what quality bar) while platform engineering owns the routing infrastructure (how policy is loaded, cached, evaluated, and observed). That boundary is what makes dynamic adjustment safe.

Reference Architecture

// TypeScript: TTL-cached routing policy loader for an async pipeline processor
import { S3Client, GetObjectCommand } from '@aws-sdk/client-s3';

interface RoutingPolicy {
  version: string;
  updatedBy: string;
  updatedAt: string;
  primaryModel: string;
  fallbackChain: Array<{
    model: string;
    triggerConditions: {
      latencyThresholdMs?: number;
      errorCodes?: string[];
      qualityScoreBelow?: number;
    };
  }>;
  qualityGate: { minScore: number; evaluatorModel: string };
}

const POLICY_TTL_MS = 30_000; // 30s — data science team sees changes within half a minute

let cachedPolicy: RoutingPolicy | null = null;
let cacheExpiresAt = 0;

export async function getRoutingPolicy(
  s3: S3Client,
  bucket: string,
  key: string
): Promise<RoutingPolicy> {
  const now = Date.now();
  if (cachedPolicy && now < cacheExpiresAt) return cachedPolicy;

  const obj = await s3.send(new GetObjectCommand({ Bucket: bucket, Key: key }));
  const body = await obj.Body?.transformToString('utf-8');
  if (!body) throw new Error('Empty routing policy object');

  cachedPolicy = JSON.parse(body) as RoutingPolicy;
  cacheExpiresAt = now + POLICY_TTL_MS;

  console.log(`[routing] Policy loaded: ${cachedPolicy.version} by ${cachedPolicy.updatedBy}`);
  return cachedPolicy;
}

State Interaction Chart

flowchart TD A[Pipeline Request] --> B[Routing Policy Loader] B --> C{Cache Hit?} C -- Yes --> D[Cached Policy] C -- No --> E[Policy Store - DB/S3] E --> D D --> F[Route Evaluator] F --> G{Primary Model OK?} G -- Yes --> H[Primary LLM] G -- No --> I{Fallback Tier 1?} I -- Yes --> J[Fallback LLM A] I -- No --> K[Fallback LLM B] H --> L[Response] J --> L K --> L L --> M[Observability Sink]

2. World-State Tracking as a Fallback Trigger: Learning from Coherent Long-Form Generation

Google's multi-agent video framework solves identity drift across shots using world-state tracking — the same mechanism is exactly what your LLM routing layer needs to detect when a model's output is drifting from expected behavior before the failure is catastrophic.

Cascading failures start small: in a sequential agent pipeline, a slightly degraded primary model response compounds with each downstream step — catching drift early via a lightweight quality signal is far cheaper than retrying the full chain.
Closed-loop refinement means the system compares each output against a maintained world-state (prior context, expected schema, consistency constraints) and triggers a fallback based on divergence score, not just HTTP error codes.
Decoupling consistency checks from the generation step itself (as Google's VQQA component does) lets you run your quality evaluator on a cheaper, faster model without slowing the primary inference path.

TECHNICAL DEEP DIVE & CODE ARTIFACTS Expand [+]

Google's framework separates the generation concern (Veo producing video frames) from the consistency concern (VQQA evaluating whether the new frame matches the established world state). The routing signal isn't 'did the API return 500' — it's 'does this output fit the maintained narrative context.' That's a much richer trigger for fallback. In an LLM pipeline serving agent tasks, your equivalent is maintaining a lightweight representation of what the task state should look like at each step, and running a fast structural check against the model's output before passing it downstream. If the output fails that check, you don't retry the same model — you route to the fallback tier with the divergence metadata attached so you can observe which conditions trigger which paths. Source: [Automating coherent long-form video generation] — Google Research (https://research.google/blog/coherent-long-form-video-generation/)

AIThis pattern connects directly to the governance trend around decomposed observability. If your fallback trigger is just 'model returned an error,' your observability surface is thin — you see the symptom but not the cause. If your trigger is a divergence score from a world-state check, every fallback event carries diagnostic payload: what the expected state was, what the model returned, and how far off it was. That data is what lets your data science team tune the policy thresholds over time — lowering the quality gate when a cheaper model is performing well, raising it when drift patterns emerge in production traffic.

Reference Architecture

// TypeScript: World-state-aware fallback trigger in a pipeline step
interface WorldState {
  expectedSchema: Record<string, unknown>;
  priorContext: string;
  stepId: string;
}

interface QualityResult {
  score: number; // 0.0 - 1.0
  reasons: string[];
}

async function routeWithDriftCheck(
  taskContext: string,
  worldState: WorldState,
  policy: RoutingPolicy,
  invokeModel: (model: string, prompt: string) => Promise<string>,
  evaluateQuality: (output: string, state: WorldState) => Promise<QualityResult>,
  emit: (event: Record<string, unknown>) => void
): Promise<string> {
  const primaryOutput = await invokeModel(policy.primaryModel, taskContext);
  const quality = await evaluateQuality(primaryOutput, worldState);

  emit({
    stepId: worldState.stepId,
    model: policy.primaryModel,
    qualityScore: quality.score,
    driftDetected: quality.score < policy.qualityGate.minScore,
    reasons: quality.reasons,
    policyVersion: policy.version,
  });

  if (quality.score >= policy.qualityGate.minScore) return primaryOutput;

  // Fallback: attach divergence context so the fallback model has signal
  const fallback = policy.fallbackChain[0];
  const enrichedPrompt = `${taskContext}\n\n[DRIFT CONTEXT]\nPrior output failed quality check (score: ${quality.score}). Issues: ${quality.reasons.join(', ')}. Correct course.`;
  return invokeModel(fallback.model, enrichedPrompt);
}

State Interaction Chart

sequenceDiagram participant P as Pipeline Step participant M as Primary LLM participant WS as World State participant QE as Quality Evaluator participant FB as Fallback LLM participant OBS as Observability P->>M: Generate(task_context) M-->>P: candidate_output P->>WS: GetExpectedState(step_id) WS-->>P: expected_schema + prior_context P->>QE: Score(candidate_output, expected_state) QE-->>P: quality_score alt quality_score >= threshold P->>OBS: Log(step, model=primary, score) P-->>P: Accept output else quality_score < threshold P->>OBS: Log(step, model=primary, score, DRIFT) P->>FB: Generate(task_context + divergence_hint) FB-->>P: fallback_output P->>OBS: Log(step, model=fallback, score) end

3. Governed Semantic Layers as the Model for Composable Routing Domains

S&P Global's pattern of giving domain experts curated, scoped agents that a proxy layer composes for cross-domain queries is a direct architectural blueprint for how you assign routing ownership across teams without losing central governance.

Domain ownership without fragmentation: each team (data science, product, operations) owns routing rules for their model domain, but a central proxy layer enforces quality gates, audit logging, and fallback ordering before any request reaches an LLM.
FastMCP-style proxy composition means your routing layer isn't a monolith — it aggregates domain-scoped routing policies at request time, which is how you support cross-domain agent tasks without collapsing all fallback logic into a single unmaintainable config.
Unity Catalog governance maps to routing policy governance: just as S&P Global uses catalog-level controls to ensure only trusted data surfaces in agent responses, your routing layer should enforce model-level controls (approved models, version constraints, cost caps) before dispatch.

TECHNICAL DEEP DIVE & CODE ARTIFACTS Expand [+]

The architectural move S&P Global made is separating the 'who curates what' concern from the 'how is it composed and governed' concern. Domain experts configure focused Genie agents per dataset without writing code; the FastMCP proxy handles cross-domain composition and the Unity Catalog layer enforces what data those agents are allowed to touch. In model routing terms, this translates to: data science teams own the routing policy for their model domain (thresholds, fallback order, quality criteria), platform engineering owns the routing infrastructure and the governance envelope (what models are approved, what cost budgets apply, what audit events are emitted). Neither team needs the other to make changes within their lane. Source: [From Data to Dialogue: How S&P Global Energy Made Its Structured Data Estate Conversational with Databricks Genie Agents and MCP] — Databricks (https://www.databricks.com/blog/data-dialogue-how-sp-global-energy-made-its-structured-data-estate-conversational-databricks)

AIThe trend toward decomposed observability connects here: if each domain has its own routing policy, you get per-domain observability for free. You can answer 'which team's model configuration caused the most fallback events last week' without building custom attribution logic. That observability feedback loop is what allows data science teams to tune their policies confidently — they see exactly which thresholds are triggering fallbacks in their domain, not in someone else's.

Reference Architecture

// TypeScript: Domain-scoped routing policy registry with central governance envelope
type ModelDomain = 'commodities' | 'financials' | 'risk';

interface DomainRoutingPolicy {
  domain: ModelDomain;
  primaryModel: string;
  fallbackChain: string[];
  qualityThreshold: number;
  costCapPerRequestUsd: number;
  ownedBy: string; // data science team identifier
}

const APPROVED_MODELS = new Set([
  'gpt-4o', 'gpt-4o-mini', 'claude-3-5-sonnet', 'claude-3-haiku',
]);

function enforceGovernanceEnvelope(
  policy: DomainRoutingPolicy
): { approved: boolean; violations: string[] } {
  const violations: string[] = [];

  if (!APPROVED_MODELS.has(policy.primaryModel)) {
    violations.push(`Primary model '${policy.primaryModel}' not in approved list`);
  }
  for (const model of policy.fallbackChain) {
    if (!APPROVED_MODELS.has(model)) {
      violations.push(`Fallback model '${model}' not in approved list`);
    }
  }
  if (policy.costCapPerRequestUsd > 0.10) {
    violations.push(`Cost cap $${policy.costCapPerRequestUsd} exceeds platform limit $0.10`);
  }

  return { approved: violations.length === 0, violations };
}

// Data science team submits a policy — governance runs before it's activated
function activateDomainPolicy(
  registry: Map<ModelDomain, DomainRoutingPolicy>,
  candidate: DomainRoutingPolicy,
  audit: (event: Record<string, unknown>) => void
): void {
  const { approved, violations } = enforceGovernanceEnvelope(candidate);
  audit({ domain: candidate.domain, ownedBy: candidate.ownedBy, approved, violations, ts: new Date().toISOString() });
  if (!approved) throw new Error(`Policy rejected: ${violations.join('; ')}`);
  registry.set(candidate.domain, candidate);
}

State Interaction Chart

flowchart TD A[Incoming Agent Request] --> B[Central Routing Proxy] B --> C{Domain Classifier} C -- Domain A --> D[Data Science Policy A] C -- Domain B --> E[Data Science Policy B] C -- Cross-Domain --> F[Composite Policy Resolver] F --> D F --> E D --> G[Governance Envelope] E --> G G --> H{Model Approved?} H -- Yes --> I[Model Dispatch] H -- No --> J[Policy Violation - Escalate] I --> K[Audit Log + Metrics] J --> K

4. Credential Abstraction as a Routing Governance Pattern

The same reason you shouldn't hand agents raw database credentials is why you shouldn't let pipeline steps know which LLM endpoint they're calling — abstraction at the routing layer is what keeps your governance enforceable and your blast radius contained.

Direct endpoint access creates a governance gap: if a pipeline step holds a model API key and a hardcoded endpoint URL, your routing policy is advisory at best — the step can bypass fallback logic entirely by accident or misconfiguration.
Routing as a capability broker means the pipeline declares what kind of task it needs handled (e.g., 'structured extraction, JSON output, max 2s latency') and the broker resolves that to a specific model + endpoint given current conditions and policy.
Audit trails require centralization: you can only answer 'which model was called for this agent task at this timestamp' if the routing layer is the single dispatch point — distributed direct calls make that reconstruction impractical.

TECHNICAL DEEP DIVE & CODE ARTIFACTS Expand [+]

CrewAI's point that the agent loop is 1% of the work while the surrounding infrastructure is 99% is easy to dismiss as founder hyperbole, but the database credential argument earns it some weight. The problem with giving agents direct data access isn't just security — it's that you lose the ability to observe, govern, and adjust behavior at the access boundary. The same logic applies to model endpoints. If your Node.js async processor holds an OpenAI API key and constructs its own HTTP calls, you've made the routing layer optional — teams will route around it when it's inconvenient, which is exactly when you need it most. Source: [Stop giving your agents database credentials] — CrewAI (https://blog.crewai.com/stop-giving-your-agents-database-credentials/)

AIThe pattern that connects the database credential argument to model routing is 'capability broker with enforced indirection.' Your routing layer should be the only code that knows model endpoint URLs and holds API keys. Pipeline steps request a capability (structured reasoning, fast summarization, code generation) from the broker, which dispatches to the appropriate model per current policy. This is what makes fallback transparent to the caller — the step doesn't know or care whether it got GPT-4o or Claude Haiku; it gets a response that met the quality bar or a clear error signal. And because every dispatch goes through the broker, your observability and audit coverage is complete by construction.

Reference Architecture

// TypeScript: Model capability broker — pipeline steps never hold credentials or endpoint URLs
type Capability = 'structured-extraction' | 'fast-summarization' | 'code-generation' | 'reasoning';

interface CapabilityRequest {
  capability: Capability;
  input: string;
  maxLatencyMs: number;
  callerStepId: string;
}

interface BrokerResponse {
  output: string;
  resolvedModel: string;
  latencyMs: number;
  fallbackUsed: boolean;
  policyVersion: string;
}

// The broker is the ONLY place that knows about model endpoints and holds credentials
class ModelCapabilityBroker {
  constructor(
    private readonly policyLoader: () => Promise<RoutingPolicy>,
    private readonly credentialProvider: (model: string) => Promise<{ apiKey: string; endpoint: string }>,
    private readonly audit: (event: Record<string, unknown>) => void
  ) {}

  async dispatch(req: CapabilityRequest): Promise<BrokerResponse> {
    const policy = await this.policyLoader();
    const start = Date.now();
    let fallbackUsed = false;

    for (const [i, modelId] of [policy.primaryModel, ...policy.fallbackChain.map(f => f.model)].entries()) {
      try {
        const creds = await this.credentialProvider(modelId);
        const output = await this.callModel(creds, req.input, req.maxLatencyMs);
        const latencyMs = Date.now() - start;
        this.audit({ stepId: req.callerStepId, model: modelId, latencyMs, fallbackUsed, policy: policy.version });
        return { output, resolvedModel: modelId, latencyMs, fallbackUsed, policyVersion: policy.version };
      } catch (err) {
        fallbackUsed = true;
        this.audit({ stepId: req.callerStepId, model: modelId, error: String(err), fallbackTier: i });
      }
    }
    throw new Error(`All models exhausted for capability: ${req.capability}`);
  }

  private async callModel(
    creds: { apiKey: string; endpoint: string },
    input: string,
    timeoutMs: number
  ): Promise<string> {
    // HTTP call to model endpoint — implementation detail hidden from callers
    const controller = new AbortController();
    const timeout = setTimeout(() => controller.abort(), timeoutMs);
    try {
      const res = await fetch(creds.endpoint, {
        method: 'POST',
        headers: { Authorization: `Bearer ${creds.apiKey}`, 'Content-Type': 'application/json' },
        body: JSON.stringify({ input }),
        signal: controller.signal,
      });
      if (!res.ok) throw new Error(`Model returned ${res.status}`);
      return (await res.json()).output as string;
    } finally {
      clearTimeout(timeout);
    }
  }
}

State Interaction Chart

flowchart TD A[Pipeline Step] -- capability request --> B[Model Capability Broker] B --> C[Policy Resolver] C --> D{Select Model} D --> E[Credential Vault] E --> F[Model API Call] F -- success --> G[Response to Step] F -- error/timeout --> H[Fallback Resolver] H --> I[Next Policy Tier] I --> E F --> J[Audit Log] H --> J

5. Operator-Facing Override Controls: Giving Data Science Teams a Real-Time Steering Interface

A routing policy that data science teams can only change by submitting a PR isn't dynamic — build a purpose-shaped control surface that makes policy tuning a first-class, audited workflow instead of an out-of-band workaround.

Canvas-style interfaces show that the right UI for agentic control is shaped around the operator's workflow, not around the system's internal structure — a routing dashboard should expose 'fallback threshold for domain X' not 'edit JSON config file Y'.
Real-time policy preview lets a data science engineer see what routing decisions the current policy would make against a sample of recent production traffic before promoting a change — this is the safeguard that makes dynamic adjustment safe.
Escalation paths matter here: when a routing change causes unexpected behavior, the system should surface that anomaly to a human reviewer with enough context (which policy version, which domain, what quality scores changed) to make a fast, informed decision.

TECHNICAL DEEP DIVE & CODE ARTIFACTS Expand [+]

GitHub's Copilot canvas pattern is interesting not for the specific Copilot use case, but for the principle it demonstrates: the interface should conform to the workflow, not the other way around. For model routing, the data science team's workflow is 'I want to understand why fallbacks are happening in my domain and adjust the threshold.' The UI for that isn't a YAML file in a git repo — it's a dashboard that shows per-domain fallback rates, per-model quality score distributions, and a single control to update the threshold with an instant preview of how that would have affected the last hour of production traffic. That workflow-shaped interface is what turns 'dynamic routing policy' from a platform promise into a daily operational tool. Source: [GitHub Copilot app for Beginners: How to build custom workflows with canvases] — GitHub (https://github.blog/ai-and-ml/github-copilot/github-copilot-app-for-beginners-how-to-build-custom-workflows-with-canvases/)

AIThis connects back to the governance theme across today's articles: governance only works if it's usable. If the control surface for routing policy is too friction-heavy, teams route around it — they hardcode endpoints, they skip the broker, they build their own retry logic. A well-designed operator interface makes the governed path the path of least resistance. The audit trail and escalation mechanisms are enforcement by default, not obstacles to work around. That's the same insight behind S&P Global giving domain experts a no-code interface to configure their Genie agents — reducing friction on the governed path is itself a governance strategy.

Reference Architecture

// TypeScript: Policy preview endpoint — simulate routing decisions against recent production events
import type { Request, Response } from 'express';

interface PolicyPreviewRequest {
  candidatePolicy: RoutingPolicy;
  domain: ModelDomain;
  lookbackWindowMinutes: number;
}

interface PolicyPreviewResult {
  totalRequests: number;
  primaryModelHits: number;
  fallbackTier1Hits: number;
  fallbackTier2Hits: number;
  exhaustedCount: number;
  estimatedCostDeltaUsd: number;
  sampleDivergentCases: Array<{ stepId: string; qualityScore: number; selectedModel: string }>;
}

async function previewRoutingPolicy(
  req: Request<{}, {}, PolicyPreviewRequest>,
  res: Response<PolicyPreviewResult>
): Promise<void> {
  const { candidatePolicy, domain, lookbackWindowMinutes } = req.body;

  // Pull recent routing events from observability store (not re-running models — replay decisions)
  const recentEvents = await fetchRoutingEvents(domain, lookbackWindowMinutes);

  let primaryHits = 0, tier1Hits = 0, tier2Hits = 0, exhausted = 0;
  const divergent: PolicyPreviewResult['sampleDivergentCases'] = [];

  for (const event of recentEvents) {
    // Replay routing decision with candidate policy thresholds against recorded quality scores
    if (event.qualityScore >= candidatePolicy.qualityGate.minScore) {
      primaryHits++;
    } else if (event.qualityScore >= candidatePolicy.qualityGate.minScore * 0.8) {
      tier1Hits++;
      if (divergent.length < 5) divergent.push({ stepId: event.stepId, qualityScore: event.qualityScore, selectedModel: candidatePolicy.fallbackChain[0]?.model ?? 'none' });
    } else if (candidatePolicy.fallbackChain.length > 1) {
      tier2Hits++;
    } else {
      exhausted++;
    }
  }

  res.json({
    totalRequests: recentEvents.length,
    primaryModelHits: primaryHits,
    fallbackTier1Hits: tier1Hits,
    fallbackTier2Hits: tier2Hits,
    exhaustedCount: exhausted,
    estimatedCostDeltaUsd: (tier1Hits * 0.002 + tier2Hits * 0.001) - (recentEvents.length * 0.003),
    sampleDivergentCases: divergent,
  });
}

// Stub — replace with your actual observability query (e.g., PostgreSQL, ClickHouse, CloudWatch Insights)
async function fetchRoutingEvents(
  domain: ModelDomain,
  lookbackMinutes: number
): Promise<Array<{ stepId: string; qualityScore: number; modelUsed: string }>> {
  return []; // query your routing audit log here
}

State Interaction Chart

flowchart TD A[Data Science Engineer] --> B[Routing Policy Dashboard] B --> C[Domain Fallback Rate Chart] B --> D[Quality Score Distribution] B --> E[Threshold Adjustment Control] E --> F[Policy Preview Engine] F --> G[Simulate Against Recent Traffic] G --> H{Looks Good?} H -- Yes --> I[Promote Policy] H -- No --> E I --> J[Policy Store] J --> K[Pipeline Picks Up on TTL Refresh] I --> L[Audit Log - who changed what when] K --> M[Observability - anomaly detection] M -- anomaly --> N[Human Review Escalation]

6. Container Platform Maturity as the Stability Floor for Dynamic LLM Routing

Dynamic routing policy is only as reliable as the container platform running your routing broker — bursty LLM traffic patterns need the same absorb-and-scale guarantees that Kubernetes provides for batch workloads, and that capability is now table stakes.

Bursty LLM demand doesn't smooth out the way API traffic does — when a primary model goes degraded and your fallback tier absorbs the overflow, your routing broker needs to scale horizontally without human intervention or it becomes the new bottleneck.
Security and governance by default at the platform layer means your routing broker's secrets (model API keys, policy store credentials) should be managed by the container platform's secrets infrastructure, not injected as environment variables by a deploy script.
Cloud-agnostic routing infrastructure means your broker should run identically on EKS, GKE, or AKS — if your fallback routing depends on an AWS-specific service, you've tightly coupled your resilience strategy to a single provider's availability.

TECHNICAL DEEP DIVE & CODE ARTIFACTS Expand [+]

AWS's fourth consecutive Gartner container management leadership recognition is a market signal more than a technical insight, but what's worth noting is the framing in the announcement: containers now run 'fleets of AI agents,' not just microservices and batch jobs. That language reflects a real operational reality — the routing broker described in earlier blocks is a stateless, horizontally scalable service that fits naturally into a container platform with HPA and KEDA-style event-driven scaling. The platform underneath your routing layer needs to handle the spike when a primary model goes down and your fallback tier gets 10x the normal traffic in 30 seconds. That's not a new scaling problem, but it's a new trigger for the scaling event. Source: [AWS named a Leader in the 2026 Gartner Magic Quadrant for Container Management] — AWS (https://aws.amazon.com/blogs/containers/aws-named-a-leader-in-the-2026-gartner-magic-quadrant-for-container-management/)

AIThe cloud-agnostic angle matters specifically for routing infrastructure. If your model broker is tightly coupled to Amazon Bedrock's routing features or Azure AI's load balancer, you've traded flexibility for convenience — and when a provider has an outage, your fallback strategy is constrained by the same blast radius. A routing broker built as a portable container service that reads policy from an object store and dispatches over standard HTTP to any model endpoint gives you genuine cross-cloud fallback capability, not just cross-model fallback. That's the architectural property worth designing for.

Reference Architecture

# Kubernetes HPA for the model routing broker
# Scales on custom metric: routing_queue_depth (fallback pressure indicator)
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: model-routing-broker-hpa
  namespace: ai-platform
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: model-routing-broker
  minReplicas: 2
  maxReplicas: 20
  metrics:
    - type: Pods
      pods:
        metric:
          name: routing_queue_depth  # emitted by broker when fallback tier is under load
        target:
          type: AverageValue
          averageValue: "10"  # scale out when avg queue depth per pod exceeds 10 requests
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 15  # fast scale-up for fallback spike absorption
      policies:
        - type: Pods
          value: 4
          periodSeconds: 15
    scaleDown:
      stabilizationWindowSeconds: 300  # slow scale-down to avoid thrashing after spike

State Interaction Chart

flowchart TD A[Load Balancer] --> B[Routing Broker - Pod 1] A --> C[Routing Broker - Pod 2] A --> D[Routing Broker - Pod N] B --> E[Policy Store - S3 or GCS] C --> E D --> E B --> F[Secrets Manager] C --> F D --> F F --> G[Model API Keys] B --> H[Primary LLM - Provider A] C --> H D --> I[Fallback LLM - Provider B] J[HPA / KEDA] -- scale on queue depth --> B J -- scale on queue depth --> C J -- scale on queue depth --> D B --> K[Metrics - Prometheus] C --> K D --> K

7. Recursive Self-Improvement as a Long-Range Risk Signal for Routing Policy Design

As models with self-modification capabilities mature, your routing policy layer needs a hard constraint against automatically promoting outputs from a self-improving model into your primary tier without human review — build that gate now before the capability arrives in production.

Recursive self-improvement loops create a routing edge case that current policy designs don't account for: what happens when the model your policy routes to is capable of modifying the weights or instructions of the model it's being routed alongside?
Policy immutability for self-modifying tiers means any model with RSI-adjacent capabilities (fine-tuning its own outputs, self-prompting, recursive instruction generation) should be firewalled to a non-promotable fallback tier until a human has reviewed a representative sample of its outputs.
Behavioral fingerprinting — tracking the statistical signature of a model's output distribution over time — is the practical early-warning system here; if the distribution shifts beyond a threshold, the routing policy should demote that model automatically pending review.

TECHNICAL DEEP DIVE & CODE ARTIFACTS Expand [+]

Lilian Weng's survey of recursive self-improvement traces the lineage from I.J. Good's ultraintelligent machine through Yudkowsky's feedback loop framing to modern systems where a model can influence its own future behavior. For production routing, this is less an immediate concern and more a design constraint to build in now. If your routing architecture allows automatic promotion (e.g., a model that consistently achieves high quality scores gets promoted from fallback tier to primary), you need an explicit carve-out: no automatic promotion for models that exhibit self-referential output patterns or that have been granted fine-tuning access to their own inference stack. The routing policy should have a 'requires_human_review_for_promotion' flag per model entry, and the promotion workflow should be a human-in-the-loop step, not an automated one. Source: [Harness Engineering for Self-Improvement] — Lilian Weng (https://lilianweng.github.io/posts/2026-07-04-harness/)

AIThis connects to the governance trend around self-modifying pipelines. Your routing policy is itself a form of pipeline configuration that agents could theoretically influence — if an agent can write to your policy store, it can change its own routing behavior. The mitigation is the same as for RSI: enforce that routing policy changes go through the same human-reviewed promotion workflow regardless of whether a human or an agent initiated the change request. The routing layer's governance envelope should be write-protected from agent-accessible code paths by construction.

Reference Architecture

// TypeScript: Routing policy model entry with RSI flag and promotion gate
interface ModelRegistryEntry {
  modelId: string;
  tier: 'primary' | 'fallback-1' | 'fallback-2' | 'quarantined';
  requiresHumanReviewForPromotion: boolean; // true for any model with self-modification capabilities
  behavioralFingerprint?: {
    outputDistributionHash: string; // rolling hash of output token distribution
    capturedAt: string;
    driftThreshold: number; // if KL divergence from baseline exceeds this, demote and alert
  };
  promotionWorkflow?: 'auto' | 'human-in-the-loop';
}

function evaluateForPromotion(
  entry: ModelRegistryEntry,
  recentQualityScores: number[],
  promotionThreshold: number,
  requestHumanReview: (modelId: string, reason: string) => void,
  promoteModel: (modelId: string) => void
): void {
  const avgScore = recentQualityScores.reduce((a, b) => a + b, 0) / recentQualityScores.length;

  if (avgScore < promotionThreshold) return; // not ready

  if (entry.requiresHumanReviewForPromotion || entry.promotionWorkflow === 'human-in-the-loop') {
    requestHumanReview(
      entry.modelId,
      `Model ${entry.modelId} reached avg quality ${avgScore.toFixed(3)} but requires human review before promotion (RSI flag: ${entry.requiresHumanReviewForPromotion})`
    );
    return;
  }

  // Only auto-promote if explicitly flagged as safe to do so
  promoteModel(entry.modelId);
}

State Interaction Chart

flowchart TD A[Model Performance Monitor] --> B{Quality Score Trend} B -- Improving --> C{Has RSI Flag?} C -- No --> D[Auto-Promote Candidate] C -- Yes --> E[Human Review Queue] D --> F[Routing Policy Store] E --> G[Human Reviewer] G -- Approved --> F G -- Rejected --> H[Keep in Fallback Tier] B -- Degrading --> I[Auto-Demote to Fallback] I --> F F --> J[Pipeline Routing Broker] J --> K[Audit Log]