Rolling Back Without Breaking the Run: CI/CD Strategies for Agentic Systems That Can't Afford to Lose State

The medical coding pipeline described in the AWS architecture post is a clean illustration of what safe model rollout looks like structurally. Step Functions owns the DAG — which step fires, in what order, and what constitutes a valid transition. Bedrock agents reason within individual steps, translating free-text clinical terms into structured codes, but they don't decide what comes next. That separation means a model rollback touches only the reasoning surface inside a step, not the workflow topology that determines how far a run has progressed. Source: [Automate medical coding with Step Functions and Amazon Bedrock AgentCore] — AWS Architecture Blog (https://aws.amazon.com/blogs/architecture/automate-medical-coding-with-step-functions-and-amazon-bedrock-agentcore/)

AIThe pattern generalizes: any agentic system where the model influences control flow — deciding which tool to call next, whether to loop, or when to escalate — creates a version coupling problem. When you roll back the model, you're not just changing inference behavior; you're potentially changing the execution path for every in-flight task. Externalizing that control into a deterministic orchestrator (whether Step Functions, Temporal, or a stateful LangGraph supervisor with explicit edges) gives your rollback a clean blast radius.

// TypeScript: model version pinned per workflow execution, not globally
interface WorkflowExecutionConfig {
  executionId: string;
  modelVersion: string;       // pinned at dispatch time
  agentSteps: AgentStepDef[];
  checkpointBucket: string;   // S3 prefix for state snapshots
}

async function dispatchWorkflow(
  config: WorkflowExecutionConfig
): Promise<void> {
  // Model version travels with the execution context —
  // a global rollback doesn't touch in-flight runs
  await sfnClient.startExecution({
    stateMachineArn: process.env.STATE_MACHINE_ARN!,
    input: JSON.stringify({
      executionId: config.executionId,
      modelVersion: config.modelVersion,  // locked at start
      checkpointBucket: config.checkpointBucket,
    }),
  });
}

// Inside each Step Functions task handler:
async function runAgentStep(
  event: { modelVersion: string; stepInput: unknown }
): Promise<unknown> {
  const client = getBedrockClient(event.modelVersion); // version-scoped client
  return client.invoke(event.stepInput);
}

AIThe real versioning surface in an agentic system isn't the model weights — it's the shape of the state that flows between steps. When you upgrade a model and it starts producing richer intermediate outputs (more structured tool call results, additional reasoning traces, expanded entity lists), those additions silently become load-bearing if downstream steps start consuming them. A rollback to the previous model then produces state that's missing fields the next step has learned to depend on. The fix is to treat your checkpoint schema the way you'd treat a database migration: every field addition is additive, every removal is deprecated-then-dropped across two releases, and your step handlers always declare which fields they require versus which they optionally consume.

AIOperationally, this means your CI pipeline for an agentic system should include a schema compatibility check — run the new model version against a corpus of saved checkpoints from production and assert that the output state is backward-readable by the previous version's step handlers. This is analogous to contract testing in microservices, and it's the layer most platform teams skip when they think of 'testing a model upgrade.' The governance dividend is that you now have a concrete, auditable artifact — the checkpoint schema diff — that you can require human sign-off on before promoting a model version to production.

// TypeScript: additive-only checkpoint schema with version tag
interface AgentCheckpoint {
  schemaVersion: number;          // bump on any structural change
  stepId: string;
  modelVersion: string;           // which model wrote this checkpoint
  requiredFields: {
    extractedEntities: Entity[];  // v1.0+
    mappedCodes: CodeResult[];    // v1.0+
  };
  optionalFields?: Record<string, unknown>; // v1.2+ additions land here
}

function readCheckpointSafely(
  raw: unknown,
  expectedSchema: number
): AgentCheckpoint {
  const parsed = AgentCheckpointSchema.parse(raw); // zod schema
  if (parsed.schemaVersion > expectedSchema) {
    // newer checkpoint — strip unknown optional fields, proceed
    return { ...parsed, optionalFields: undefined };
  }
  return parsed;
}

// Contract test: assert v1.1 handler can read v1.2 checkpoint
test('v1.1 step handler tolerates v1.2 checkpoint', () => {
  const checkpoint = buildCheckpointWithV12Fields();
  const result = readCheckpointSafely(checkpoint, /* v1.1 schema */ 1);
  expect(result.requiredFields.mappedCodes).toBeDefined();
});

Grafana Workflows makes a pointed observation: the coordination work during an incident is itself a job, and it competes for attention with the diagnostic work. That tension is worse in agentic systems because model degradation is subtler than a service going down — it shows up as drift in output distributions, increases in human reviewer override rates, or downstream tasks quietly producing wrong results. By the time it's obvious, the blast radius is already large. Source: [Spend less time on operational busywork and resolve incidents faster with Grafana Workflows] — Grafana Labs (https://grafana.com/blog/spend-less-time-on-operational-busywork-and-resolve-incidents-faster-with-grafana-workflows/)

AIThe architectural implication for agentic CI/CD is that model rollback needs to be a first-class incident type in your observability stack, not an afterthought in your deployment playbook. That means instrumenting your agent steps to emit the metrics that would trigger a rollback — not just latency and error rate, but semantic signals like code mapping rejection rate, schema conformance percentage, or downstream human override frequency. Wire those into your alerting layer with defined thresholds, and your rollback runbook becomes executable, not just documented.

// TypeScript: emit semantic rollback trigger metrics from agent step handler
async function executeAgentStep(
  stepCtx: StepContext
): Promise<StepResult> {
  const result = await invokeAgent(stepCtx);

  // Structural validation against expected output schema
  const schemaValid = OutputSchema.safeParse(result.output);
  
  await metrics.putMetric({
    namespace: 'AgentOps/ModelHealth',
    dimensions: { modelVersion: stepCtx.modelVersion, stepId: stepCtx.stepId },
    metrics: [
      { name: 'SchemaConformanceRate', value: schemaValid.success ? 1 : 0, unit: 'Count' },
      { name: 'OutputConfidenceScore', value: result.confidenceScore, unit: 'None' },
    ],
  });

  if (!schemaValid.success) {
    // Don't silently continue — surface the violation
    await alertingClient.triggerAnomaly({
      type: 'model-output-schema-violation',
      modelVersion: stepCtx.modelVersion,
      executionId: stepCtx.executionId,
      details: schemaValid.error.format(),
    });
  }

  return result;
}

Modal's architecture — and the thesis behind their Series D — is that developers shouldn't need to pre-provision capacity or predict load to run inference workloads. For model rollouts, this matters because the economics of canary deployments change completely: instead of reserving a fraction of your existing GPU fleet for a canary lane (and eating the cost whether the canary is active or not), you dispatch the canary to serverless compute that only runs when it's handling requests. Source: [Why AI Compute Comes in Rectangles — Erik Bernhardsson, Modal CEO] — Latent Space (https://www.latent.space/p/modal-d)

AIFor Sam's cloud-agnostic infrastructure mandate, this pattern is worth operationalizing at the platform layer: abstract your agent invocation behind a model router that resolves version-to-endpoint at dispatch time, and let the endpoint be a serverless function on whatever provider best fits the workload. The rollback becomes a config change in the router — swap the canary endpoint back to the stable endpoint — and the infrastructure cost of maintaining both versions in parallel is near zero because serverless compute scales to zero when idle. The governance dividend is that you can keep the previous model version live as a rollback target indefinitely without paying for standby capacity.

// TypeScript: version-aware model router with canary split
interface ModelRoute {
  stableVersion: string;
  canaryVersion?: string;
  canaryTrafficPercent?: number;  // 0-100
}

async function resolveModelEndpoint(
  route: ModelRoute,
  executionId: string
): Promise<string> {
  if (
    route.canaryVersion &&
    route.canaryTrafficPercent &&
    shouldRouteToCanary(executionId, route.canaryTrafficPercent)
  ) {
    // Deterministic per-execution: same run always hits same version
    return getServerlessEndpoint(route.canaryVersion);
  }
  return getServerlessEndpoint(route.stableVersion);
}

function shouldRouteToCanary(
  executionId: string,
  percent: number
): boolean {
  // Hash-based: execution is sticky to a version for its lifetime
  const hash = murmurhash(executionId) % 100;
  return hash < percent;
}

Amazon Polly's generative engine upgrade is marketed as a drop-in replacement — same voice IDs, improved output, no API changes required. That's a clean story for a TTS use case where the output is audio and quality is subjective. Source: [Amazon Polly delivers a major upgrade to its Generative engine] — AWS Machine Learning Blog (https://aws.amazon.com/blogs/machine-learning/amazon-polly-delivers-a-major-upgrade-to-its-generative-engine/). For agentic systems where model outputs feed downstream structured processing — entity extraction, code mapping, decision routing — a 'drop-in' upgrade that changes output phrasing, confidence calibration, or structured field population can break your pipeline in ways that don't surface as errors, they surface as wrong answers.

AIThe governance pattern here is to treat provider-initiated model upgrades the same way you'd treat a dependency bumping a minor version that touches your critical path. That means maintaining a pinned version in your invocation config, subscribing to provider release notes as a signal to trigger your eval pipeline, and having a documented process for accepting or deferring an upgrade. The recursive self-improvement framing from Lilian Weng's harness post adds a sharper edge to this concern: if provider models are themselves getting better at producing training signal from their own outputs, the pace of provider-side upgrades is likely to accelerate, which means your governance process for accepting those upgrades needs to be lightweight enough to keep up. Source: [Harness Engineering for Self-Improvement] — Lilian Weng (https://lilianweng.github.io/posts/2026-07-04-harness/)

// TypeScript: explicit model version pinning with upgrade governance
const MODEL_VERSION_CONFIG = {
  // Never rely on 'latest' — pin explicitly, upgrade deliberately
  textExtraction: 'anthropic.claude-3-5-sonnet-20241022-v2:0',
  codeMapping:    'anthropic.claude-3-5-haiku-20241022-v1:0',
} as const;

type ModelRole = keyof typeof MODEL_VERSION_CONFIG;

async function invokeWithPinnedVersion(
  role: ModelRole,
  prompt: string
): Promise<ModelOutput> {
  const modelId = MODEL_VERSION_CONFIG[role];
  // modelId is a literal type — a provider upgrade requires
  // a deliberate code change, not a config swap
  return bedrockClient.invoke({ modelId, prompt });
}

// In your eval harness: assert output schema stability across versions
async function assertUpgradeSafe(
  prevVersion: string,
  newVersion: string,
  corpus: EvalCase[]
): Promise<CompatibilityReport> {
  const prevOutputs = await runCorpus(prevVersion, corpus);
  const newOutputs  = await runCorpus(newVersion, corpus);
  return compareOutputDistributions(prevOutputs, newOutputs);
}