Checkpointing Agent State at Failure Boundaries: Patterns for Long-Running Tasks That Don't Start Over

The core design decision is granularity: too coarse and you replay expensive LLM calls or irreversible side effects; too fine and your checkpoint overhead dominates task runtime. Phase boundaries — where a discrete unit of agent work has produced a committable result — solve this naturally. Each phase checkpoint should carry: the phase identifier, input context hash, output payload, tool call records with their responses, and a status enum (PENDING, COMPLETED, FAILED, SKIPPED). Writing these as a single PostgreSQL transaction means your checkpoint is either fully durable or not at all — no torn writes.

The partial-credit scoring approach in cybersecurity eval frameworks makes this concrete: an agent working an attack chain (recon → exploit → exfiltrate) is evaluated on which phases completed, not just whether the final goal was reached. That same model maps directly onto checkpoint design — each phase is independently verifiable, independently resumable, and independently auditable. Source: [Patterns for Building Cybersecurity Evals] — Eugene Yan (https://eugeneyan.com//writing/cybersecurity-evals/)

// TypeScript — Phase checkpoint writer for PostgreSQL
import { Pool } from 'pg';

type PhaseStatus = 'PENDING' | 'COMPLETED' | 'FAILED' | 'SKIPPED';

interface PhaseCheckpoint {
  taskId: string;
  phaseId: string;
  inputHash: string;
  outputPayload: Record<string, unknown>;
  toolCallLog: { tool: string; input: unknown; output: unknown }[];
  status: PhaseStatus;
  committedAt?: Date;
}

async function commitPhaseCheckpoint(
  pool: Pool,
  checkpoint: PhaseCheckpoint
): Promise<void> {
  await pool.query(
    `INSERT INTO agent_phase_checkpoints
       (task_id, phase_id, input_hash, output_payload, tool_call_log, status, committed_at)
     VALUES ($1, $2, $3, $4, $5, $6, NOW())
     ON CONFLICT (task_id, phase_id)
     DO UPDATE SET
       output_payload = EXCLUDED.output_payload,
       tool_call_log  = EXCLUDED.tool_call_log,
       status         = EXCLUDED.status,
       committed_at   = NOW()
     WHERE agent_phase_checkpoints.status != 'COMPLETED'`,
    [
      checkpoint.taskId,
      checkpoint.phaseId,
      checkpoint.inputHash,
      JSON.stringify(checkpoint.outputPayload),
      JSON.stringify(checkpoint.toolCallLog),
      checkpoint.status,
    ]
  );
}

async function resumeFromLastCheckpoint(
  pool: Pool,
  taskId: string
): Promise<string> {
  const result = await pool.query(
    `SELECT phase_id FROM agent_phase_checkpoints
     WHERE task_id = $1 AND status = 'COMPLETED'
     ORDER BY committed_at DESC LIMIT 1`,
    [taskId]
  );
  return result.rows[0]?.phase_id ?? 'TASK_START';
}

The 'Ticket Triage Triangle' problem — incomplete fields, slow handoffs, eventing gaps — is a production-grade description of what happens when a multi-step agentic workflow lacks durable state emission at each transition. In a CrewAI-based triage system, each agent handoff is a potential checkpoint boundary: the intake agent completes enrichment, emits a structured event with all resolved fields, and only then does the routing agent consume it. If that event is missing or partial, you don't just lose observability — you lose the anchor point that makes a safe resume possible. Source: [How a Mid-Tier Enterprise SaaS Provider Automates Cloud Support Triage] — CrewAI Blog (https://blog.crewai.com/mid-tier-enterprise-saas-automates-cloud-support-triage/)

AIThe connection between eventing discipline and checkpoint integrity runs deeper than it looks: an event schema that enforces completeness (required fields, type validation at emission time) is also a checkpoint schema that enforces recoverability. Teams that treat state emission as optional telemetry consistently discover the gap when they try to build resume logic and find they have no reliable record of which phases actually committed. Design the event schema first, then derive both the checkpoint write and the observability stream from it.

// TypeScript — Enforced event schema at handoff boundaries
import { z } from 'zod';

const IntakeCompleteEvent = z.object({
  taskId:        z.string().uuid(),
  phaseId:       z.literal('INTAKE_COMPLETE'),
  ticketId:      z.string(),
  severity:      z.enum(['P1', 'P2', 'P3', 'P4']),
  affectedService: z.string().min(1),
  enrichedFields: z.record(z.string()),
  committedAt:   z.string().datetime(),
});

type IntakeCompleteEvent = z.infer<typeof IntakeCompleteEvent>;

async function emitHandoffEvent(
  eventStore: EventStore,
  raw: unknown
): Promise<void> {
  // Validation failure here is a hard stop — not a swallowed error.
  // A partial event is worse than no event: it gives false resume confidence.
  const event = IntakeCompleteEvent.parse(raw);
  await eventStore.append(event.taskId, event);
}

Eugene Yan's cybersecurity eval framework lays out a four-component structure — sandboxed targets, difficulty-graded inputs, tool access, and outcome graders — that maps surprisingly cleanly onto a checkpoint test harness. The sandboxed target is your isolated agent runtime. The grader that scores partial attack-chain completion is your checkpoint completeness validator. The key insight is that 'did the agent reach the final goal' is the wrong question for both security evals and checkpoint testing; the right question is 'which phases completed cleanly and what state did each one produce?' Source: [Patterns for Building Cybersecurity Evals] — Eugene Yan (https://eugeneyan.com//writing/cybersecurity-evals/)

AIThis reframes checkpoint testing as a form of eval discipline: you're not just checking that the task succeeded, you're asserting invariants about the state machine — that completed phases are idempotent under replay, that failed phases leave no partial writes, and that the resume path from any checkpoint produces identical downstream behavior to a clean run. Teams shipping production agents without this discipline are flying without instruments — they discover checkpoint corruption at 2am when a long-running task fails on retry instead of recovering.

// TypeScript — Checkpoint integrity test using fault injection
import { describe, it, expect, vi } from 'vitest';

describe('Phase checkpoint resume integrity', () => {
  it('resumes from failed phase without replaying completed phases', async () => {
    const toolCallSpy = vi.fn();
    const agent = buildTestAgent({ onToolCall: toolCallSpy });

    // Run until phase 3 fails
    await agent.run('task-001', { injectFailureAtPhase: 'PHASE_3' });

    const checkpoints = await db.getCheckpoints('task-001');
    expect(checkpoints['PHASE_1'].status).toBe('COMPLETED');
    expect(checkpoints['PHASE_2'].status).toBe('COMPLETED');
    expect(checkpoints['PHASE_3'].status).toBe('FAILED');

    const callsBeforeResume = toolCallSpy.mock.calls.length;

    // Resume — should pick up at PHASE_3, not PHASE_1
    await agent.resume('task-001');

    // Only PHASE_3 tools should have fired again
    const newCalls = toolCallSpy.mock.calls.slice(callsBeforeResume);
    const replayedPhases = newCalls.map(([call]) => call.phase);
    expect(replayedPhases).not.toContain('PHASE_1');
    expect(replayedPhases).not.toContain('PHASE_2');
  });
});

The practical problem is that a long-running agent accumulates context across dozens of tool calls, sub-agent outputs, and intermediate reasoning steps — and a naive checkpoint that serializes the full conversation history becomes expensive to store, slow to load, and likely to hit context limits on resume. The solution is a layered context model: persist the phase output payload (structured, compact, schema-validated) as the checkpoint, and reconstruct the context window at resume time by composing the system scaffold with a summarized phase history. This is the same problem senior engineers face in long Claude Code sessions — context drift and window pressure require deliberate pruning of what gets carried forward. Source: [How building software is changing at Anthropic] — Pragmatic Engineer (https://newsletter.pragmaticengineer.com/p/inside-anthropic)

AIThe key architectural principle here is that your checkpoint should store the agent's conclusions, not its conversation. A phase that spent 8 LLM calls determining that a ticket needs P2 routing should checkpoint 'severity: P2, routing_target: platform-team' — not 8 turns of reasoning. Resuming with that compact state is faster, cheaper, and produces more reliable downstream behavior than replaying the full reasoning chain from a serialized transcript.

// TypeScript — Layered context reconstruction on resume
interface PhaseOutput {
  phaseId: string;
  summary: string;        // LLM-generated compression of phase reasoning
  structuredResult: Record<string, unknown>;
}

async function reconstructContextOnResume(
  systemScaffold: string,
  taskId: string,
  db: CheckpointStore
): Promise<string> {
  const completedPhases: PhaseOutput[] = await db.getCompletedPhases(taskId);

  const phaseHistory = completedPhases
    .map(p => `[${p.phaseId}] ${p.summary}\nResult: ${JSON.stringify(p.structuredResult)}`)
    .join('\n\n');

  // Compose scaffold + compressed history — never replay raw conversation
  return [
    systemScaffold,
    '--- Completed phases ---',
    phaseHistory,
    '--- Resume from next phase ---',
  ].join('\n');
}

// On resume, agent gets clean context without megabytes of prior turns
const context = await reconstructContextOnResume(SYSTEM_PROMPT, 'task-001', db);

NorthStar's build story — a mobile-friendly React/TypeScript scheduling app for 3,000 clinicians delivered in weeks — illustrates the gap between delivery speed and operational depth. The platform gave them velocity; the hard question is whether the state model underneath is designed to recover cleanly when a scheduling agent fails mid-run with 1,500 clinicians' time-off data partially written. That's not a criticism of the delivery approach — it's a structural observation about what durable execution discipline looks like when you're moving fast. Source: [How NorthStar Anesthesia built a scheduling app for a workforce of 3,000 clinicians in weeks] — Databricks Blog (https://www.databricks.com/blog/how-northstar-anesthesia-built-scheduling-app-workforce-3000-clinicians-weeks)

AIThe pattern across all the production deployments in today's digest is that state management debt accumulates silently during fast builds and surfaces noisily at scale. The mitigation isn't to slow down — it's to treat checkpoint schema design as a day-one architectural constraint rather than a post-launch concern. Fifteen minutes defining your PhaseCheckpoint type and your idempotency key strategy at the start of a sprint is worth more than a week of incident response after the first partial-state corruption in production.

// TypeScript — Schema-first checkpoint type definition (day one, before agent logic)
import { z } from 'zod';

// Define this before writing any agent logic.
// Checkpoint schema IS your state contract.
const SchedulingPhaseCheckpoint = z.object({
  taskId:          z.string().uuid(),
  phaseId:         z.enum(['FETCH_TIMEOFF', 'RESOLVE_CONFLICTS', 'WRITE_SCHEDULE', 'NOTIFY_CLINICIANS']),
  idempotencyKey:  z.string(), // hash(taskId + phaseId + inputHash)
  cliniciansProcessed: z.array(z.string()),
  outputPayload:   z.record(z.unknown()),
  status:          z.enum(['PENDING', 'COMPLETED', 'FAILED']),
  committedAt:     z.string().datetime().optional(),
});

export type SchedulingPhaseCheckpoint = z.infer<typeof SchedulingPhaseCheckpoint>;

// Idempotency key prevents duplicate writes on retry
function buildIdempotencyKey(taskId: string, phaseId: string, inputHash: string): string {
  return `${taskId}::${phaseId}::${inputHash}`;
}