1. Idempotent Checkpoint Writes as the Foundation of Restartable Agents

Before you design any checkpointing strategy, every state write must be idempotent — otherwise a restart after a partial write leaves you with corrupted state that looks valid until it quietly breaks downstream steps.

  • Checkpoint granularity determines your blast radius on failure — too coarse and you replay expensive LLM calls; too fine and your checkpoint overhead dominates the actual work.
  • PostgreSQL UPSERT semantics give you exactly the idempotency guarantee you need for checkpoint writes: a checkpoint keyed by (workflow_id, step_sequence_number) either inserts cleanly or overwrites a duplicate without side effects.
  • Step sequence numbers should be monotonically increasing and assigned before the step executes, not after — this is the difference between 'I know this step ran' and 'I know this step was supposed to run.'
TECHNICAL DEEP DIVE & CODE ARTIFACTS Expand [+]

The core of a restartable agent is a state table where each row captures the input, output, tool call results, and LLM response for a single logical step — not the entire workflow state as a blob. When a workflow restarts, it queries for the highest committed sequence number under its workflow ID and resumes from the next step, replaying nothing that already has a committed row. This design keeps your restart logic dead simple: no diffing, no merging, just a MAX(step_seq) query and a cursor forward.

The trap most teams fall into is checkpointing the full in-memory agent state as a serialized object. That approach creates implicit coupling between your checkpoint format and your agent code — a schema change in your state model means old checkpoints are unreadable. Prefer a narrow, stable checkpoint schema that captures inputs and outputs at the tool-call boundary, and reconstruct agent context in memory from those records on restart. Your checkpoint table becomes an append-only event log, which also hands you provenance for free.

Reference Architecture
// TypeScript — minimal checkpoint writer for a step-based agent workflow
import { Pool } from 'pg';

interface StepCheckpoint {
  workflowId: string;
  stepSeq: number;
  stepName: string;
  inputHash: string;       // SHA-256 of serialized step input
  outputPayload: unknown;  // raw tool/LLM output
  completedAt: Date;
}

async function writeCheckpoint(
  pool: Pool,
  checkpoint: StepCheckpoint
): Promise<void> {
  await pool.query(
    `INSERT INTO agent_checkpoints
       (workflow_id, step_seq, step_name, input_hash, output_payload, completed_at)
     VALUES ($1, $2, $3, $4, $5, $6)
     ON CONFLICT (workflow_id, step_seq)
     DO UPDATE SET
       output_payload = EXCLUDED.output_payload,
       completed_at   = EXCLUDED.completed_at`,
    [
      checkpoint.workflowId,
      checkpoint.stepSeq,
      checkpoint.stepName,
      checkpoint.inputHash,
      JSON.stringify(checkpoint.outputPayload),
      checkpoint.completedAt,
    ]
  );
}

async function findResumePoint(
  pool: Pool,
  workflowId: string
): Promise<number> {
  const result = await pool.query<{ max_step_seq: number | null }>(
    `SELECT MAX(step_seq) AS max_step_seq
     FROM agent_checkpoints
     WHERE workflow_id = $1`,
    [workflowId]
  );
  return (result.rows[0].max_step_seq ?? -1) + 1;
}
State Interaction Chart
flowchart TD A[Workflow Start] --> B{Checkpoint Exists?} B -- Yes --> C[Load Max Step Seq] B -- No --> D[Initialize Step Seq = 0] C --> E[Resume from Next Step] D --> F[Execute Step 0] E --> G[Execute Next Step] F --> H[Write Checkpoint Row] G --> H H --> I{More Steps?} I -- Yes --> G I -- No --> J[Mark Workflow Complete] H -- Write Fails --> K[Retry Idempotent Write] K --> I

2. Git-Inspired Branching State for Concurrent and Speculative Agent Execution

GitHub's work scaling Git for millions of concurrent agent commits is a direct architectural signal: treating agent state snapshots like commits — immutable, addressable, and branch-able — solves the concurrent multi-agent state collision problem that flat checkpoint tables cannot.

  • Immutable snapshot references let you run speculative agent branches (exploring two tool-call strategies in parallel) and then merge or discard without ever corrupting the main workflow state.
  • Parent-child checkpoint links give you a DAG of state evolution that doubles as your audit trail — you can walk backwards from any failure to the exact state that preceded it, without separate logging infrastructure.
  • Concurrent agent writes to shared state are safe only when each agent owns a distinct state branch; merging at a supervisor node is explicit and auditable rather than implicit and racy.
TECHNICAL DEEP DIVE & CODE ARTIFACTS Expand [+]

GitHub's infrastructure team is rebuilding Git's underlying architecture to handle workloads where agents are committing at the same rate humans once did — millions of operations per day against the same repositories. The key insight they're working with is that concurrency at agent scale breaks assumptions baked into Git's original locking model. The same pressure exists inside multi-agent state stores: if three subagents are executing in parallel and all writing to a shared workflow state object, you have a distributed write conflict problem that no amount of application-level locking fully solves.

The cleaner architectural response is to give each agent its own append-only state branch, keyed by (workflow_id, agent_id, branch_id), and promote state upward to the supervisor only at explicit merge points. This is structurally identical to a feature branch workflow — agents work in isolation, a supervisor node reviews and merges their outputs, and the canonical workflow state only advances when a merge is committed. It's heavier to implement than a single shared state table, but it gives you safe parallelism, rollback capability, and a built-in audit log without extra instrumentation. Source: [Building Git infrastructure for agent-scale development] — GitHub Engineering (https://github.blog/engineering/architecture-optimization/building-git-infrastructure-for-agent-scale-development/)

Reference Architecture
// TypeScript — branched checkpoint structure for parallel subagents
interface AgentStateBranch {
  workflowId: string;
  agentId: string;
  branchId: string;         // UUID per parallel execution attempt
  parentBranchId: string | null;  // null for root branch
  stepSeq: number;
  stateSnapshot: unknown;
  mergedToCanonical: boolean;
  createdAt: Date;
}

// Supervisor merge: promote a branch's final state to canonical
async function mergeBranchToCanonical(
  pool: Pool,
  workflowId: string,
  branchId: string
): Promise<void> {
  await pool.query(
    `WITH branch_final AS (
       SELECT state_snapshot
       FROM agent_state_branches
       WHERE workflow_id = $1 AND branch_id = $2
       ORDER BY step_seq DESC
       LIMIT 1
     )
     UPDATE workflow_canonical_state
     SET state_snapshot    = branch_final.state_snapshot,
         last_merged_branch = $2,
         updated_at         = NOW()
     FROM branch_final
     WHERE workflow_id = $1`,
    [workflowId, branchId]
  );
  await pool.query(
    `UPDATE agent_state_branches
     SET merged_to_canonical = true
     WHERE workflow_id = $1 AND branch_id = $2`,
    [workflowId, branchId]
  );
}
State Interaction Chart
flowchart TD A[Supervisor Node] --> B[Branch: Agent A State] A --> C[Branch: Agent B State] A --> D[Branch: Agent C State] B --> E[Agent A Checkpoints] C --> F[Agent B Checkpoints] D --> G[Agent C Checkpoints] E --> H[Merge Point: Supervisor Review] F --> H G --> H H --> I{Merge OK?} I -- Yes --> J[Advance Canonical Workflow State] I -- No --> K[Discard Branch / Retry] K --> A

3. Distributed Tracing as Checkpoint Verification Infrastructure

Grafana Tempo 3.1's sampling-aware TraceQL metrics mean you can now query across traces to verify that your agent's checkpoints are landing where and when they should — turning your tracing backend into a passive test harness for checkpoint fidelity.

  • Trace spans bracketing checkpoint writes give you wall-clock evidence that a checkpoint completed before the next step started — without this, you cannot distinguish 'step completed but checkpoint failed' from 'step did not complete' during incident investigation.
  • Query-based trace redaction in Tempo 3.1 matters for agent workflows that checkpoint sensitive tool call outputs — you can strip PII from trace payloads at query time without losing the structural timing data you need for debugging.
  • Sampling-aware metrics solve the problem of high-frequency checkpointing drowning your trace backend — you get accurate aggregate latency and error rates across millions of checkpoint spans without storing every individual trace.
TECHNICAL DEEP DIVE & CODE ARTIFACTS Expand [+]

The connection between distributed tracing and checkpoint verification is underused in agentic systems. Most teams instrument their LLM calls and tool invocations but treat checkpoint writes as internal bookkeeping — invisible to the trace. That's a mistake. A checkpoint write should be a child span of the step it's sealing, with attributes carrying the workflow_id, step_seq, and a boolean confirming the write succeeded. When a workflow fails and you're diagnosing whether the last checkpoint was committed, you want to answer that question from your trace backend — not from a database query that requires reconstructing the sequence of events.

Tempo 3.1's sampling-aware TraceQL metrics change the calculus for high-throughput agentic pipelines. If you're running hundreds of concurrent workflows each writing checkpoints every few seconds, naive trace storage would collapse under the volume. Sampling-aware metrics let you record statistical summaries of checkpoint write latency and failure rate across the full population of traces, while retaining full detail only for anomalous ones. That's the right tradeoff: you keep aggregate visibility without paying full storage costs, and you still get complete traces for every checkpoint failure. Source: [Tempo 3.1 release: new features for Kafka, TraceQL metrics updates, trace redaction, and more] — Grafana Labs (https://grafana.com/blog/tempo-3-1-release-all-the-latest-features/)

Reference Architecture
// TypeScript — wrapping checkpoint writes in OpenTelemetry spans
import { trace, SpanStatusCode } from '@opentelemetry/api';

const tracer = trace.getTracer('agent-workflow');

async function executeStepWithCheckpoint(
  pool: Pool,
  workflowId: string,
  stepSeq: number,
  stepFn: () => Promise<unknown>
): Promise<void> {
  await tracer.startActiveSpan(`execute_step`, async (stepSpan) => {
    stepSpan.setAttributes({ 'workflow.id': workflowId, 'step.seq': stepSeq });
    try {
      const output = await stepFn();

      await tracer.startActiveSpan('write_checkpoint', async (cpSpan) => {
        try {
          await writeCheckpoint(pool, {
            workflowId,
            stepSeq,
            stepName: `step_${stepSeq}`,
            inputHash: '',
            outputPayload: output,
            completedAt: new Date(),
          });
          cpSpan.setAttributes({ 'checkpoint.success': true });
        } catch (err) {
          cpSpan.setStatus({ code: SpanStatusCode.ERROR, message: String(err) });
          cpSpan.setAttributes({ 'checkpoint.success': false });
          throw err;
        } finally {
          cpSpan.end();
        }
      });
    } finally {
      stepSpan.end();
    }
  });
}
State Interaction Chart
sequenceDiagram participant Agent as Agent Step Executor participant CP as Checkpoint Writer participant DB as PostgreSQL participant Tempo as Grafana Tempo Agent->>Tempo: Start span: execute_step(step_seq=N) Agent->>Agent: Run tool call / LLM inference Agent->>Tempo: Start child span: write_checkpoint(step_seq=N) Agent->>CP: writeCheckpoint(workflowId, stepSeq, output) CP->>DB: UPSERT agent_checkpoints DB-->>CP: OK CP-->>Agent: success=true Agent->>Tempo: End child span (checkpoint.success=true) Agent->>Tempo: End parent span Tempo-->>Tempo: Apply sampling policy Tempo-->>Tempo: Compute TraceQL metrics aggregate

4. Sovereignty-Aware State Storage: Where Agent Checkpoints Can and Cannot Live

The AWS Digital Sovereignty Lens is a forcing function for agentic teams in regulated industries: if your agent checkpoints sensitive data — tool call outputs, inferred user intent, retrieved documents — then the residency and auditability of that state is a compliance requirement, not an infrastructure preference.

  • Checkpoint data residency must be defined at workflow design time, not retrofitted after your first compliance audit — the schema needs columns for data_classification, storage_region, and retention_policy from day one.
  • Sensitive tool call outputs serialized into checkpoint rows are the quiet compliance risk in most agentic systems — if your agent retrieves PII during a task and checkpoints that retrieval result, that PII is now sitting in your state store with whatever retention policy you applied to workflow metadata.
  • Sovereignty requirements decompose across workflow phases — the LLM inference call might be subject to one jurisdiction, the tool call output to another, and the final workflow result to a third; your checkpoint schema needs to reflect those boundaries.
TECHNICAL DEEP DIVE & CODE ARTIFACTS Expand [+]

AWS's Well-Architected Sovereignty Lens formalizes something agentic engineers have been improvising around: that data residency, encryption key ownership, and audit trail completeness are architectural constraints that must be expressed in system design, not just in deployment configuration. For agent state stores specifically, this means that a single PostgreSQL table holding all workflow checkpoints may violate sovereignty requirements if workflows processing data from different jurisdictions share the same physical table — even if the data is encrypted at rest.

The practical response is a checkpoint routing layer that directs state writes to region-specific stores based on the workflow's data classification metadata, established before any step executes. This isn't exotic — it's the same pattern as multi-region data residency in any enterprise SaaS product, applied to the checkpoint store. The added complexity is that agentic workflows are often long-running and stateful, so a residency decision made at workflow initialization must be enforced consistently across every subsequent checkpoint write for the lifetime of that workflow. Building that enforcement into your checkpoint writer — rather than relying on application code scattered across agent implementations — is the architectural discipline that makes it auditable. Source: [Announcing the AWS Digital Sovereignty Lens for the Well-Architected Framework] — AWS Architecture Blog (https://aws.amazon.com/blogs/architecture/announcing-the-aws-digital-sovereignty-well-architected-lens/)

Reference Architecture
// TypeScript — checkpoint router enforcing data residency at write time
type DataClassification = 'public' | 'internal' | 'sensitive' | 'regulated';

interface WorkflowMeta {
  workflowId: string;
  classification: DataClassification;
  requiredRegion: string;  // e.g. 'eu-west-1', 'us-east-1'
}

class SovereignCheckpointWriter {
  private storeRegistry: Map<string, Pool>;

  constructor(storeRegistry: Map<string, Pool>) {
    this.storeRegistry = storeRegistry;
  }

  async write(
    meta: WorkflowMeta,
    checkpoint: Omit<StepCheckpoint, 'workflowId'>
  ): Promise<void> {
    const store = this.resolveStore(meta);
    await writeCheckpoint(store, {
      ...checkpoint,
      workflowId: meta.workflowId,
    });
    await this.emitAuditEvent(meta, checkpoint.stepSeq);
  }

  private resolveStore(meta: WorkflowMeta): Pool {
    const store = this.storeRegistry.get(meta.requiredRegion);
    if (!store) {
      throw new Error(
        `No checkpoint store registered for region ${meta.requiredRegion} ` +
        `(required by workflow ${meta.workflowId}, classification: ${meta.classification})`
      );
    }
    return store;
  }

  private async emitAuditEvent(meta: WorkflowMeta, stepSeq: number): Promise<void> {
    // emit to your audit trail — CloudWatch, Kafka, etc.
    console.log(JSON.stringify({
      event: 'checkpoint_written',
      workflowId: meta.workflowId,
      stepSeq,
      region: meta.requiredRegion,
      classification: meta.classification,
      timestamp: new Date().toISOString(),
    }));
  }
}
State Interaction Chart
flowchart TD A[Workflow Init] --> B[Assign Data Classification] B --> C{Classification?} C -- Sensitive/Regulated --> D[Route to Region-Specific Store] C -- Internal --> E[Route to Default Store] D --> F[Checkpoint Writer: enforce region] E --> F F --> G[Write Checkpoint with residency metadata] G --> H[Audit Log Entry] H --> I[Step Execution Continues]

5. Container Resource Isolation as a Checkpoint Reliability Dependency

cgroup v2's unified memory pressure model means your agent container can now receive accurate, early signals before it's OOM-killed — which gives you a narrow but real window to flush a checkpoint before the process dies and you lose everything since the last committed state.

  • Memory pressure notifications in cgroup v2 (via the memory.pressure interface) let your agent process detect impending resource exhaustion and trigger an emergency checkpoint flush before the kernel kills the container.
  • CPU throttling visibility under cgroup v2 is significantly more granular than v1 — for long-running agent workflows, unexpected CPU throttling is often the leading indicator of a cascade that ends in an OOM or timeout failure.
  • Kubernetes resource limits on agent pods must be sized with checkpoint write overhead in mind — a PostgreSQL UPSERT under memory pressure is slower than under normal conditions, and if your limit is too tight, the emergency checkpoint flush itself may not complete.
TECHNICAL DEEP DIVE & CODE ARTIFACTS Expand [+]

Kubernetes v1.31 moving cgroup v1 into maintenance mode is an infrastructure forcing function that most agentic platform teams haven't fully internalized yet. The shift to cgroup v2 isn't just an OS-level detail — it changes the resource visibility your agent processes have access to, and those signals are directly relevant to checkpoint reliability. Under cgroup v2's unified hierarchy, you can use the memory.pressure PSI (Pressure Stall Information) file to register callbacks when memory pressure crosses a threshold, giving your agent runtime a chance to write a defensive checkpoint before the kernel decides to reclaim memory forcibly. This is the difference between a clean restart from the last checkpoint and a full workflow replay from scratch.

The practical implementation is a lightweight watchdog thread (or a Node.js worker thread if you're running your agent in a JS runtime) that polls memory pressure and invokes a checkpoint flush if pressure exceeds a configured threshold. It's not zero cost — you're adding async I/O under conditions where you're already memory-constrained — but the alternative is accepting that any OOM event resets your workflow to the beginning. For agents running 30-minute tasks, that's an unacceptable tradeoff. Sizing your Kubernetes resource limits to include headroom for at least one checkpoint write under pressure conditions is the kind of detail that separates a reliably restartable agent from one that theoretically checkpoints but practically loses state on every real failure. Source: [The Shift to cgroup v2 in Kubernetes: What You Need to Know] — Kubernetes.io (https://kubernetes.io/blog/2026/10/06/kubernetes-cgroups-v2-shift/)

Reference Architecture
// TypeScript — Node.js worker thread watching cgroup v2 memory pressure
import { workerData, parentPort } from 'worker_threads';
import { readFileSync } from 'fs';

const PRESSURE_FILE = '/sys/fs/cgroup/memory.pressure';
const POLL_INTERVAL_MS = 2_000;
const SOME_THRESHOLD_PCT = 40; // tune for your workload

function parsePressureSome(content: string): number {
  // Format: some avg10=X.XX avg60=X.XX avg300=X.XX total=N
  const match = content.match(/some avg10=(\d+\.\d+)/);
  return match ? parseFloat(match[1]) : 0;
}

function watchMemoryPressure(): void {
  setInterval(() => {
    try {
      const raw = readFileSync(PRESSURE_FILE, 'utf-8');
      const someAvg10 = parsePressureSome(raw);
      if (someAvg10 > SOME_THRESHOLD_PCT) {
        parentPort?.postMessage({
          event: 'memory_pressure_high',
          someAvg10,
          action: 'flush_checkpoint',
        });
      }
    } catch {
      // cgroup v2 not available — degrade gracefully
    }
  }, POLL_INTERVAL_MS);
}

watchMemoryPressure();
State Interaction Chart
flowchart TD A[Agent Container Running] --> B[cgroup v2 PSI Watchdog] B --> C{Memory Pressure > Threshold?} C -- No --> D[Continue Normal Execution] D --> B C -- Yes --> E[Trigger Emergency Checkpoint Flush] E --> F{Checkpoint Write Succeeds?} F -- Yes --> G[Log: safe shutdown point established] F -- No --> H[Log: flush failed - accept potential state loss] G --> I[Container Continues or Exits Gracefully] H --> I

6. Enterprise Agentic Continuity: State Handoff Across Sessions and User Contexts

The Atlassian-OpenAI integration is a production signal that enterprise agents must maintain coherent state not just across failures, but across intentional session boundaries — a user resuming a Jira-driven workflow tomorrow needs the agent to pick up where it left off, not re-derive its context from scratch.

  • Session-boundary checkpoints are architecturally distinct from failure-recovery checkpoints — they must serialize enough human-readable context for a human reviewer (or a supervisor agent) to validate that the resumed state is still correct given any changes in the external environment.
  • External knowledge graph drift is the enterprise-specific failure mode: an agent checkpoints its understanding of a Confluence page at 9am, and that page is edited at 2pm before the user resumes the workflow — the agent's checkpoint is stale but not invalid, and your system needs a strategy for detecting and handling that.
  • Workflow versioning across checkpoints becomes essential at enterprise scale — if your agent's tool definitions or prompt templates change between when a workflow was checkpointed and when it resumes, replaying from checkpoint may produce different behavior than the original run.
TECHNICAL DEEP DIVE & CODE ARTIFACTS Expand [+]

AIThe Atlassian and OpenAI partnership expanding into enterprise knowledge action surfaces a class of checkpointing challenges that pure infrastructure reliability work doesn't address: the semantics of a checkpoint across time and human context change. A failure-recovery checkpoint assumes the world is roughly the same when you restart as it was when you paused. An enterprise session-boundary checkpoint cannot make that assumption — documents get updated, tickets get resolved, people change their minds. What you need isn't just a snapshot of agent state, but a snapshot plus a validity envelope: a set of conditions under which that checkpoint is still trustworthy as a resume point.

Practically, this means your checkpoint schema should include a staleness_check field: a list of external resource identifiers and their last-known versions (document IDs and ETags, ticket states, etc.) that the agent was reasoning about when it checkpointed. On resume, before replaying any steps, your workflow runner queries those resources for current versions and compares. If anything has changed beyond a configured tolerance, the workflow is flagged for human review before continuing — not automatically restarted. This is human-in-the-loop design applied at the checkpoint layer, not just at escalation thresholds. It's a direct extension of the governance checkpoint pattern, and it's the kind of detail that makes an enterprise agent trustworthy rather than merely functional. Source: [Atlassian and OpenAI expand partnership to turn enterprise knowledge into action] — OpenAI (https://openai.com/index/atlassian-partnership)

Reference Architecture
// TypeScript — checkpoint with staleness envelope for enterprise session resumption
interface ResourceVersion {
  resourceId: string;
  resourceType: 'confluence_page' | 'jira_ticket' | 'github_pr';
  versionToken: string;  // ETag, last_modified, or revision hash
}

interface SessionCheckpoint extends StepCheckpoint {
  resourceVersions: ResourceVersion[];  // resources agent relied on
  checkpointedAt: Date;
}

async function validateCheckpointFreshness(
  checkpoint: SessionCheckpoint,
  fetchCurrentVersion: (id: string, type: string) => Promise<string>
): Promise<{ fresh: boolean; staleResources: ResourceVersion[] }> {
  const stale: ResourceVersion[] = [];

  for (const rv of checkpoint.resourceVersions) {
    const currentVersion = await fetchCurrentVersion(rv.resourceId, rv.resourceType);
    if (currentVersion !== rv.versionToken) {
      stale.push({ ...rv, versionToken: currentVersion });
    }
  }

  return { fresh: stale.length === 0, staleResources: stale };
}

// Usage at workflow resume time:
// const { fresh, staleResources } = await validateCheckpointFreshness(checkpoint, fetchVersion);
// if (!fresh) { await escalateToHumanReview(workflowId, staleResources); return; }
// await resumeWorkflowFromCheckpoint(checkpoint);
State Interaction Chart
sequenceDiagram participant User participant Supervisor as Supervisor Agent participant Store as Checkpoint Store participant Ext as External Knowledge Sources User->>Supervisor: Resume workflow Supervisor->>Store: Load checkpoint + resource versions Supervisor->>Ext: Verify current resource versions Ext-->>Supervisor: Return current ETags/states Supervisor->>Supervisor: Compare: checkpoint versions vs current alt All resources unchanged Supervisor->>Supervisor: Resume from checkpoint step else Resources changed Supervisor->>User: Flag for human review User-->>Supervisor: Approve or discard checkpoint end

7. Action Categories and the Reflexive Agent Model

Chip Huyen's framework for categorizing agent actions — read, write, execute — directly determines your checkpointing obligations: read-only actions are cheap to replay without a checkpoint, but write and execute actions have side effects that cannot be safely undone, making their pre-execution state the one you absolutely cannot lose.

  • Write and execute actions define your mandatory checkpoint boundaries — any step that mutates external state (API calls, database writes, file system changes) must be preceded by a committed checkpoint so that on restart you know with certainty whether the action was attempted.
  • Exactly-once semantics for agent tool calls with side effects require an idempotency key that travels through the checkpoint — if you restart and re-execute a tool call with the same key, the downstream system can detect and discard the duplicate.
  • Read-only action replay is almost always cheaper than checkpointing every retrieval result — design your checkpoint strategy around the action category taxonomy, not around a blanket 'checkpoint every step' rule.
TECHNICAL DEEP DIVE & CODE ARTIFACTS Expand [+]

The action categorization lens from Huyen's agent framework is one of the most practically useful tools for designing a proportionate checkpointing strategy. The instinct to checkpoint everything is understandable but produces bloated state stores and slows down hot paths. The right discipline is to ask: if this step fails and I restart, can I safely replay it? For a vector retrieval step with no side effects, the answer is almost always yes — don't checkpoint the result, just re-run it. For a step that sends an email, updates a ticket, or calls a payment API, the answer is no — you must checkpoint before execution, and you must issue the tool call with an idempotency key so a restart doesn't duplicate the action.

The idempotency key design deserves its own attention. It should be derived from the workflow_id and step_seq, not from a random UUID generated at call time — a UUID generated on first execution won't be regenerated identically on restart, defeating the purpose. A deterministic key like SHA-256(workflow_id + ':' + step_seq) gives you the same key on both the original and any retry execution, letting the downstream tool deduplicate cleanly. This is standard practice in payment systems and event-driven architectures; it's less commonly applied to agentic tool calls, but the reliability argument is identical. Source: [Agents] — Chip Huyen (https://huyenchip.com//2025/01/07/agents.html)

Reference Architecture
// TypeScript — deterministic idempotency key + pre/post checkpoint for write actions
import { createHash } from 'crypto';

function deriveIdempotencyKey(workflowId: string, stepSeq: number): string {
  return createHash('sha256')
    .update(`${workflowId}:${stepSeq}`)
    .digest('hex')
    .slice(0, 32);  // most APIs accept up to 36 chars
}

type ActionCategory = 'read' | 'write' | 'execute';

async function runStepWithGovernance<T>(
  pool: Pool,
  workflowId: string,
  stepSeq: number,
  category: ActionCategory,
  toolFn: (idempotencyKey?: string) => Promise<T>
): Promise<T> {
  const idempotencyKey = deriveIdempotencyKey(workflowId, stepSeq);

  if (category !== 'read') {
    // Mandatory pre-execution checkpoint for side-effecting actions
    await writeCheckpoint(pool, {
      workflowId,
      stepSeq,
      stepName: `pre_exec_${stepSeq}`,
      inputHash: idempotencyKey,
      outputPayload: { status: 'pre_execution', category },
      completedAt: new Date(),
    });
  }

  const result = await toolFn(category !== 'read' ? idempotencyKey : undefined);

  await writeCheckpoint(pool, {
    workflowId,
    stepSeq,
    stepName: `post_exec_${stepSeq}`,
    inputHash: idempotencyKey,
    outputPayload: result,
    completedAt: new Date(),
  });

  return result;
}
State Interaction Chart
flowchart TD A[Agent Step] --> B{Action Category?} B -- Read Only --> C[Execute Without Pre-Checkpoint] B -- Write or Execute --> D[Write Pre-Execution Checkpoint] C --> E[Cache Result if Expensive] D --> F[Derive Idempotency Key] F --> G[Execute Tool Call with Key] G --> H{Tool Call Succeeded?} H -- Yes --> I[Write Post-Execution Checkpoint] H -- No --> J{Restart? Key Matches?} J -- Yes --> K[Downstream Deduplicates] J -- No --> L[Re-execute with Same Key] K --> I L --> H