Checkpoint or Restart: Engineering Resumable Agent Workflows That Survive Real Failures

Think of an agent workflow as a series of rooms connected by doors — a checkpoint lives at the door, not somewhere in the middle of the room. When Southern Company's SCOUT integrates outage, customer, and terrain data layers into a unified operational picture, each data domain is a distinct processing phase: raw ingestion, enrichment, correlation, and presentation. If correlation fails, you don't want to re-ingest terabytes of Delta Lake data; you want to resume at the start of the correlation phase with the enriched intermediate state already materialized. That's the engineering discipline: identify your phase seams first, then design your checkpoint schema around what state a new worker actually needs to pick up cleanly at that seam. Source: [Southern Company's SCOUT: Completing the Storm Intelligence Story] — (https://www.databricks.com/blog/southern-companys-scout-completing-storm-intelligence-story)

AIThe practical implementation in a Node.js/TypeScript async processor maps naturally to a state machine where each phase transition writes a checkpoint record to PostgreSQL before yielding — the checkpoint row includes the phase name, a content-addressed snapshot of relevant intermediate state, the schema version, and a monotonic sequence number for ordering. Resumption then becomes a simple query: find the latest completed phase checkpoint for this workflow run, deserialize the state blob, and hand it to the worker for the next phase. The schema version field is what separates 'recoverable' from 'must restart' when you deploy new code mid-flight.

// TypeScript: Phase-boundary checkpoint writer for a PostgreSQL-backed agent workflow
import { Pool } from 'pg';
import { createHash } from 'crypto';

type CheckpointRecord = {
  workflowId: string;
  phase: string;
  schemaVersion: number;
  stateHash: string;
  stateBlob: unknown;
  completedAt: Date;
};

async function writePhaseCheckpoint(
  pool: Pool,
  workflowId: string,
  phase: string,
  state: unknown,
  schemaVersion = 2
): Promise<void> {
  const stateJson = JSON.stringify(state);
  const stateHash = createHash('sha256').update(stateJson).digest('hex');

  await pool.query(
    `INSERT INTO agent_checkpoints
     (workflow_id, phase, schema_version, state_hash, state_blob, completed_at)
     VALUES ($1, $2, $3, $4, $5::jsonb, NOW())
     ON CONFLICT (workflow_id, phase)
     DO UPDATE SET state_blob = EXCLUDED.state_blob,
                   state_hash = EXCLUDED.state_hash,
                   schema_version = EXCLUDED.schema_version,
                   completed_at = EXCLUDED.completed_at`,
    [workflowId, phase, schemaVersion, stateHash, stateJson]
  );
}

async function resumeFromCheckpoint(
  pool: Pool,
  workflowId: string,
  targetSchemaVersion: number
): Promise<{ phase: string; state: unknown } | null> {
  const { rows } = await pool.query(
    `SELECT phase, state_blob, schema_version
     FROM agent_checkpoints
     WHERE workflow_id = $1
       AND schema_version = $2
     ORDER BY completed_at DESC
     LIMIT 1`,
    [workflowId, targetSchemaVersion]
  );
  if (!rows.length) return null;
  return { phase: rows[0].phase, state: rows[0].state_blob };
}

The cybersecurity eval architecture Eugene Yan describes — sandboxed targets, graded inputs, a toolkit of bash and debuggers, and automated graders — is essentially a blueprint for observable long-horizon task execution. The pyramid structure (find vulnerability → reproduce it → exploit it → achieve attacker goal) encodes the intuition that progress is hierarchical and that partial completion has real value. Translated to a general agent checkpointing framework: define an ordered set of progress tiers for your task type, write a lightweight grader function for each tier, and store the achieved tier alongside your state snapshot in every checkpoint write. Source: [Patterns for Building Cybersecurity Evals] — (https://eugeneyan.com//writing/cybersecurity-evals/)

This matters most when you're running evaluations or production tasks with expensive tool calls — think satellite data processing pipelines or multi-step code generation. If your checkpoint record includes both the phase boundary (where you are structurally) and the progress tier (how far you've gotten semantically), your orchestrator can make smarter resume decisions: retry the same tier with a different tool strategy, escalate to a human reviewer, or advance to the next phase. [AI Synthesis] Storing tier achievements as a separate column from state blobs also makes your checkpoint table queryable as an observability feed — you can aggregate across workflow runs to see where agents are statistically stalling, which is the kind of signal that drives real infrastructure and prompt improvements.

// TypeScript: Progress-tier-aware checkpoint with grader integration
type ProgressTier = 0 | 1 | 2 | 3 | 4;

interface TieredCheckpoint {
  workflowId: string;
  phase: string;
  progressTier: ProgressTier;
  tierLabel: string;
  stateBlob: unknown;
  gradedAt: Date;
}

const TIER_LABELS: Record<ProgressTier, string> = {
  0: 'not_started',
  1: 'target_found',
  2: 'condition_reproduced',
  3: 'action_executed',
  4: 'goal_achieved',
};

async function gradeAndCheckpoint(
  pool: Pool,
  workflowId: string,
  phase: string,
  state: unknown,
  graders: Array<(s: unknown) => Promise<boolean>>
): Promise<ProgressTier> {
  let achievedTier: ProgressTier = 0;

  for (let tier = 1; tier <= graders.length; tier++) {
    const passed = await graders[tier - 1](state);
    if (!passed) break;
    achievedTier = tier as ProgressTier;
  }

  await pool.query(
    `INSERT INTO tiered_checkpoints
     (workflow_id, phase, progress_tier, tier_label, state_blob, graded_at)
     VALUES ($1, $2, $3, $4, $5::jsonb, NOW())
     ON CONFLICT (workflow_id, phase)
     DO UPDATE SET progress_tier = EXCLUDED.progress_tier,
                   tier_label = EXCLUDED.tier_label,
                   state_blob = EXCLUDED.state_blob,
                   graded_at = EXCLUDED.graded_at`,
    [workflowId, phase, achievedTier, TIER_LABELS[achievedTier], JSON.stringify(state)]
  );

  return achievedTier;
}

Kubernetes v1.37 graduating HPA scale-to-zero to Beta and enabling it by default means that agent worker deployments using external or object metrics can now drop to zero replicas between task phases without any add-ons or feature gates. The practical implication for checkpoint design: your checkpoint store becomes the source of truth for whether any work is in progress, and your event bus (SQS, Kafka, EventBridge) becomes the signal that wakes a worker. If you design your phase checkpoints to emit an event on write, you get scale-to-zero for free — the worker has no reason to stay alive between the checkpoint write and the next phase trigger. Source: [Kubernetes v1.37: Scale Workloads to Zero with HorizontalPodAutoscaler] — (https://kubernetes.io/blog/2026/09/02/kubernetes-v1-37-hpa-scale-to-zero-beta/)

AIThe pairing of scale-to-zero with checkpoint-driven resume also changes how you handle partial failures in regulated environments. When a Fargate pod running a checkpoint-aware agent gets evicted mid-phase, the last committed checkpoint record is your ground truth — not the worker's in-memory state. This is why writing checkpoints transactionally (checkpoint row committed before the phase result is acted upon) is the same discipline as two-phase commit in distributed systems: the checkpoint write is your durable intent log. Cold-start on Fargate under Kyverno-managed proxy injection adds another dimension to this — the proxy must be injected and connected before the worker can reach your checkpoint store or LLM API, so proxy readiness is now part of your resume path's critical chain. Source: [Automate proxy injection for Amazon EKS on AWS Fargate using Kyverno] — (https://aws.amazon.com/blogs/containers/automate-proxy-injection-for-amazon-eks-on-aws-fargate-using-kyverno/)

// TypeScript: Transactional checkpoint-then-emit pattern for scale-to-zero worker handoff
import { Pool } from 'pg';
import { SQSClient, SendMessageCommand } from '@aws-sdk/client-sqs';

async function commitCheckpointAndHandoff(
  pool: Pool,
  sqs: SQSClient,
  queueUrl: string,
  workflowId: string,
  completedPhase: string,
  nextPhase: string,
  state: unknown,
  progressTier: number
): Promise<void> {
  const client = await pool.connect();
  try {
    await client.query('BEGIN');

    // Write checkpoint inside transaction
    await client.query(
      `INSERT INTO agent_checkpoints
       (workflow_id, phase, progress_tier, state_blob, completed_at)
       VALUES ($1, $2, $3, $4::jsonb, NOW())
       ON CONFLICT (workflow_id, phase)
       DO UPDATE SET state_blob = EXCLUDED.state_blob,
                     progress_tier = EXCLUDED.progress_tier,
                     completed_at = EXCLUDED.completed_at`,
      [workflowId, completedPhase, progressTier, JSON.stringify(state)]
    );

    await client.query('COMMIT');

    // Only emit after durable commit — worker can now safely scale down
    await sqs.send(new SendMessageCommand({
      QueueUrl: queueUrl,
      MessageBody: JSON.stringify({ workflowId, nextPhase, progressTier }),
      MessageGroupId: workflowId, // FIFO queue ordering per workflow
    }));
  } catch (err) {
    await client.query('ROLLBACK');
    throw err;
  } finally {
    client.release();
  }
}

MAPL-EMIT achieves 84% recall on annotated methane plumes by processing NASA EMIT hyperspectral data through a deep learning model that turns raw satellite observations into localized, quantified emissions signals. The pipeline's structure — raw observations in, actionable stakeholder data out — is architecturally identical to a long-running agentic pipeline: expensive data ingestion, model inference per batch, and downstream delivery to consumers who act on the results. The checkpointing challenge here is that partial publication is a side effect: if you've already emitted plume detections for South Asia and the pipeline fails on Europe, resuming from scratch means either re-emitting South Asia detections (duplicates downstream) or implementing idempotency at the consumer level. Source: [Mapping global methane emissions from space with deep learning] — (https://research.google/blog/mapping-global-methane-emissions-from-space-with-deep-learning/)

AIThe general principle is that checkpoints must capture not just 'where we are' but 'what we've already committed downstream' — the emission offset, the published event IDs, the rows already written. This is the same discipline as at-least-once delivery with idempotency keys in distributed systems, applied to agent pipelines. In a PostgreSQL-backed platform, this translates to a checkpoint record that includes a delivered_artifact_ids array — every downstream delivery gets logged before the checkpoint advances, so a resumed worker can skip re-delivery for already-processed batches. This pattern connects directly to the per-phase provenance logging trend: your checkpoint store is your provenance log, and the delivered artifacts column is your delivery receipt ledger.

// TypeScript: Idempotent batch checkpoint with delivery receipt tracking
interface BatchCheckpoint {
  workflowId: string;
  batchId: string;
  region: string;
  processedAt: Date | null;
  deliveredArtifactIds: string[];
  signalQuality: 'high' | 'medium' | 'low' | 'noisy';
}

async function recordBatchDelivery(
  pool: Pool,
  workflowId: string,
  batchId: string,
  artifactId: string,
  signalQuality: BatchCheckpoint['signalQuality']
): Promise<void> {
  // Append artifactId to delivery receipt — safe to call multiple times
  await pool.query(
    `INSERT INTO batch_checkpoints
     (workflow_id, batch_id, delivered_artifact_ids, signal_quality, processed_at)
     VALUES ($1, $2, ARRAY[$3::text], $4, NOW())
     ON CONFLICT (workflow_id, batch_id)
     DO UPDATE SET
       delivered_artifact_ids = array_append(
         batch_checkpoints.delivered_artifact_ids,
         EXCLUDED.delivered_artifact_ids[1]
       ),
       signal_quality = EXCLUDED.signal_quality,
       processed_at = EXCLUDED.processed_at
     WHERE NOT ($3 = ANY(batch_checkpoints.delivered_artifact_ids))`,
    [workflowId, batchId, artifactId, signalQuality]
  );
}

async function getUnprocessedBatches(
  pool: Pool,
  workflowId: string,
  allBatchIds: string[]
): Promise<string[]> {
  const { rows } = await pool.query(
    `SELECT batch_id FROM batch_checkpoints
     WHERE workflow_id = $1
       AND processed_at IS NOT NULL`,
    [workflowId]
  );
  const processed = new Set(rows.map((r: { batch_id: string }) => r.batch_id));
  return allBatchIds.filter(id => !processed.has(id));
}

SCOUT's architecture — Delta Lake for durable storage, lakehouse warehouses for query performance, Unity Catalog for governance — maps cleanly onto the concerns of a production agent checkpoint store. Delta Lake's ACID transaction guarantees are the same property you want from your PostgreSQL checkpoint table: a write either fully committed or it didn't happen. Unity Catalog's fine-grained access control is the analog of per-phase checkpoint visibility policies — not every consumer of your agent's output should be able to read the raw intermediate state from Phase 2, especially if it contains PII or unreleased data. Source: [Southern Company's SCOUT: Completing the Storm Intelligence Story] — (https://www.databricks.com/blog/southern-companys-scout-completing-storm-intelligence-story)

AIThe deeper governance lesson from SCOUT is that checkpoint infrastructure and observability infrastructure should be the same system, not two separate concerns bolted together. When 1,139 users are querying operational state in real time, the checkpoint store isn't just a crash recovery mechanism — it's the authoritative view of workflow progress, domain health, and data freshness. Designing your agent checkpoint schema with that dual purpose in mind (recovery + observability) from the start means your platform team gets a unified dashboard for 'where are all running workflows right now' as a free side effect of robust crash recovery. This is the pattern signal the last week has been converging on: unified governance dashboards emerge naturally from well-designed checkpoint stores, not from bolt-on monitoring layers.

// TypeScript: Governance-aware checkpoint reader with access policy enforcement
type CheckpointRole = 'ops_reader' | 'resume_worker' | 'compliance_auditor' | 'admin';

const PHASE_ACCESS_POLICY: Record<string, CheckpointRole[]> = {
  'ingest':     ['resume_worker', 'admin'],
  'enrich':     ['resume_worker', 'admin'],
  'correlate':  ['resume_worker', 'ops_reader', 'admin'],
  'present':    ['resume_worker', 'ops_reader', 'compliance_auditor', 'admin'],
};

async function readCheckpointWithPolicy(
  pool: Pool,
  workflowId: string,
  phase: string,
  callerRole: CheckpointRole
): Promise<unknown | null> {
  const allowedRoles = PHASE_ACCESS_POLICY[phase];
  if (!allowedRoles?.includes(callerRole)) {
    throw new Error(
      `Role '${callerRole}' is not permitted to read checkpoint for phase '${phase}'`
    );
  }

  await pool.query(
    `INSERT INTO checkpoint_access_log (workflow_id, phase, caller_role, accessed_at)
     VALUES ($1, $2, $3, NOW())`,
    [workflowId, phase, callerRole]
  );

  const { rows } = await pool.query(
    `SELECT state_blob FROM agent_checkpoints
     WHERE workflow_id = $1 AND phase = $2
     ORDER BY completed_at DESC LIMIT 1`,
    [workflowId, phase]
  );

  return rows[0]?.state_blob ?? null;
}