Per-Phase Provenance Logging: Building Audit-Grade Observability Into Multi-Agent LLM Pipelines

The core design challenge is deciding what constitutes a phase. In a LangGraph supervisor graph, you might be tempted to log at the node level, but that conflates tool calls with actual data transformations. A better boundary is the edge — specifically, the state delta produced when control moves from one node to the next. What you want to capture is the diff between the graph state before and after a node fires, tagged with the node's identity, the LLM call that drove it (including model ID, temperature, and token counts), and a cryptographic hash of both the input state snapshot and the output state snapshot. That hash chain is what gives you tamper-evidence without requiring a blockchain.

The practical engineering pattern here is a thin provenance middleware that wraps your graph's state reducer. Every time state is committed, the middleware intercepts the write, computes the delta, serializes it to an append-only log (a PostgreSQL table with no UPDATE or DELETE grants on the service account works well), and only then allows the write to proceed. This makes provenance logging transactionally coupled to state mutation — if the log write fails, the state write rolls back, and you never have a pipeline step that ran without a record. [AI Synthesis] This pattern directly addresses the governance and auditing focus that has been accelerating across the industry, particularly as more capable models like GPT-6 Astra get deployed into agentic pipelines where the cost of an untraced decision is substantially higher.

import { createHash } from 'crypto';
import { Pool } from 'pg';

interface ProvenanceRecord {
  pipelineRunId: string;
  sequenceNumber: number;
  agentId: string;
  modelId: string;
  inputHash: string;
  outputHash: string;
  inputSnapshot: unknown;
  outputSnapshot: unknown;
  wallClockMs: number;
}

async function commitWithProvenance<S>(
  pool: Pool,
  runId: string,
  seq: number,
  agentId: string,
  modelId: string,
  inputState: S,
  outputState: S,
  applyMutation: (client: import('pg').PoolClient) => Promise<void>
): Promise<void> {
  const client = await pool.connect();
  try {
    await client.query('BEGIN');

    const inputHash = createHash('sha256')
      .update(JSON.stringify(inputState))
      .digest('hex');
    const outputHash = createHash('sha256')
      .update(JSON.stringify(outputState))
      .digest('hex');

    const record: ProvenanceRecord = {
      pipelineRunId: runId,
      sequenceNumber: seq,
      agentId,
      modelId,
      inputHash,
      outputHash,
      inputSnapshot: inputState,
      outputSnapshot: outputState,
      wallClockMs: Date.now(),
    };

    await client.query(
      `INSERT INTO agent_provenance_log
         (pipeline_run_id, seq, agent_id, model_id, input_hash, output_hash,
          input_snapshot, output_snapshot, recorded_at)
       VALUES ($1,$2,$3,$4,$5,$6,$7,$8,to_timestamp($9::bigint / 1000.0))`,
      [
        record.pipelineRunId, record.sequenceNumber, record.agentId,
        record.modelId, record.inputHash, record.outputHash,
        JSON.stringify(record.inputSnapshot), JSON.stringify(record.outputSnapshot),
        record.wallClockMs,
      ]
    );

    await applyMutation(client);
    await client.query('COMMIT');
  } catch (err) {
    await client.query('ROLLBACK');
    throw err;
  } finally {
    client.release();
  }
}

Grafana Labs documented exactly this failure mode: each new AI experiment built its own provider client, and within months the team had four different streaming abstractions, three different error models, and no consistent way to correlate a pipeline trace across experiments. Their response was to extract a shared Go SDK that normalizes the provider interface, handles streaming and tool-call protocols uniformly, and becomes the single place where observability hooks live. The lesson is not Go-specific — the structural point holds for TypeScript just as well. Source: [The Grafana AI SDK for Go: a shared foundation for building AI applications] — Grafana Labs (https://grafana.com/blog/the-grafana-ai-sdk-for-go-a-shared-foundation-for-building-ai-applications/)

AIThis maps directly onto the per-phase provenance problem: if your provenance middleware wraps a shared SDK rather than individual LLM clients, you get structural consistency in your log records for free. Every provenance entry has the same shape regardless of which team's agent produced it, which is what makes cross-pipeline audits feasible at scale. The cost of not doing this compounds fast — retrofitting consistent log schema onto six independently built clients mid-production is exactly the kind of painful migration that could have been avoided with upfront SDK discipline.

// shared-llm-client.ts — single intercept point for all agent LLM calls
import { createHash } from 'crypto';

interface LLMCallMeta {
  modelId: string;
  promptHash: string;
  completionHash: string;
  promptTokens: number;
  completionTokens: number;
  latencyMs: number;
  agentId: string;
  runId: string;
}

type ObservabilityHook = (meta: LLMCallMeta) => Promise<void>;

export function createInstrumentedClient(
  provider: (modelId: string, prompt: string) => Promise<{ text: string; promptTokens: number; completionTokens: number }>,
  hooks: ObservabilityHook[]
) {
  return async function call(modelId: string, prompt: string, agentId: string, runId: string): Promise<string> {
    const start = Date.now();
    const promptHash = createHash('sha256').update(prompt).digest('hex');

    const result = await provider(modelId, prompt);

    const meta: LLMCallMeta = {
      modelId,
      promptHash,
      completionHash: createHash('sha256').update(result.text).digest('hex'),
      promptTokens: result.promptTokens,
      completionTokens: result.completionTokens,
      latencyMs: Date.now() - start,
      agentId,
      runId,
    };

    await Promise.all(hooks.map(h => h(meta)));
    return result.text;
  };
}

GPT-6 Astra's availability on Amazon Bedrock is a meaningful escalation point: organizations are already running production agentic workflows on Bedrock, and introducing a significantly more capable reasoning model into those pipelines without revisiting the observability architecture is a mistake. The model's deeper judgment is precisely what makes traceability more valuable — when an agent makes a complex business decision driven by extended reasoning, the audit trail needs to capture not just what was decided but what context was available at decision time, including which tools were called and what they returned. Source: [Take on your most ambitious work with GPT-6 Astra on Amazon Bedrock] — AWS (https://aws.amazon.com/blogs/machine-learning/take-on-your-most-ambitious-work-with-gpt-6-astra-on-amazon-bedrock/)

AIThe practical implication for provenance schema design is that fields like `reasoning_trace`, `tool_calls_log`, and `context_window_hash` are no longer optional. A provenance record for a GPT-6 Astra decision that only contains input and output hashes is functionally incomplete for audit purposes — you cannot reconstruct the causal chain that led to the output without the intermediate artifacts. Design your schema to accommodate this now, because retrofitting it after a compliance request is a painful exercise.

// provenance schema for high-capability model decisions
interface AgentDecisionRecord {
  runId: string;
  stepId: string;
  agentId: string;
  modelId: string;           // e.g. 'gpt-6-astra'
  provider: string;          // e.g. 'bedrock'
  contextWindowHash: string; // SHA-256 of full context sent to model
  toolCallsLog: Array<{
    toolName: string;
    inputHash: string;
    outputHash: string;
    durationMs: number;
  }>;
  reasoningTraceHash: string | null;  // null if model does not expose CoT
  reasoningTraceText: string | null;
  outputHash: string;
  outputText: string;
  promptTokens: number;
  completionTokens: number;
  timestampMs: number;
  pipelineSequenceNumber: number;
}

Eugene Yan's framework for working with AI — maintaining annotated indexes, building persistent memory layers, onboarding models with project-specific documentation — is usually framed as a productivity pattern, but it maps directly onto provenance architecture when you apply it to production agents rather than personal assistants. The difference is that in production, every artifact injected into an agent's context at session start needs a version identifier and a snapshot reference, because that context is causal evidence for every decision the agent makes during that session. If you are feeding an agent a retrieved memory chunk that was written by a prior agent session, the provenance graph needs an edge from the earlier session's output to this session's input. Source: [How to Work and Compound with AI] — Eugene Yan (https://eugeneyan.com//writing/working-with-ai/)

AIThis is where the distinction between session-level and run-level provenance becomes important. A run is a single pipeline execution — a discrete unit with clear boundaries. A session spans multiple runs and accumulates state through a memory layer. Your provenance model needs both: run-level records that capture per-step transformations, and session-level records that capture how the memory layer evolved across runs. Without the session layer, you can audit any individual run but you cannot reconstruct why an agent's behavior changed over time.

// session-level provenance envelope — wraps run records
interface SessionProvenanceEnvelope {
  sessionId: string;
  startedAt: number;
  endedAt: number | null;
  memoryStoreVersionAtStart: string;    // e.g. git-style hash or monotonic version ID
  docIndexVersionAtStart: string;
  systemPromptHash: string;
  runs: string[];                        // ordered run IDs within this session
  memoryWriteEvents: Array<{
    runId: string;
    stepId: string;
    keyWritten: string;
    previousValueHash: string | null;
    newValueHash: string;
    timestampMs: number;
  }>;
}

// Usage: before dispatching any run in a session, snapshot current context versions
async function openSession(
  memoryStore: { currentVersion(): string },
  docIndex: { currentVersion(): string },
  systemPrompt: string
): Promise<SessionProvenanceEnvelope> {
  const { createHash } = await import('crypto');
  return {
    sessionId: crypto.randomUUID(),
    startedAt: Date.now(),
    endedAt: null,
    memoryStoreVersionAtStart: memoryStore.currentVersion(),
    docIndexVersionAtStart: docIndex.currentVersion(),
    systemPromptHash: createHash('sha256').update(systemPrompt).digest('hex'),
    runs: [],
    memoryWriteEvents: [],
  };
}

Claude's watermarking approach works by subtly biasing the probability distribution over tokens during generation in a way that is statistically detectable but does not measurably degrade output quality. The key property from a provenance perspective is that it survives moderate transformation — you can detect model origin even in text that has been lightly edited or reformatted, which is exactly what happens when one agent passes content to another for refinement or summarization. This gives you a secondary attribution layer that is independent of your logging infrastructure. Source: [How Claude Watermarks AI-Generated Text] — Sebastian Raschka (https://magazine.sebastianraschka.com/p/claude-watermarking)

AIIn a multi-agent pipeline context, watermark propagation is an interesting governance question: if Agent A produces watermarked content and Agent B rewrites it substantially, does the watermark survive? The answer determines whether you can use watermark detection as a lightweight cross-agent provenance check at consumption points, or whether you need full input-output hashing at every phase boundary regardless. For now, the engineering answer is both — watermark checks are fast and cheap, so run them as a first-pass attribution gate, and fall back to hash-based provenance for authoritative audit records.

// Lightweight watermark attribution checkpoint in a pipeline step
// Assumes access to a watermark detection service endpoint

async function checkWatermarkProvenance(
  content: string,
  expectedModelFamily: string,
  runId: string,
  stepId: string,
  provenanceLog: { flagMissingAttribution(runId: string, stepId: string, content: string): Promise<void> }
): Promise<{ attributed: boolean; confidence: number; modelFamily: string | null }> {
  const response = await fetch('https://internal-watermark-detector/check', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ text: content }),
  });

  const result = await response.json() as { detected: boolean; confidence: number; modelFamily: string | null };

  if (!result.detected || result.modelFamily !== expectedModelFamily) {
    // Watermark absent or wrong family — flag for full provenance audit
    await provenanceLog.flagMissingAttribution(runId, stepId, content);
  }

  return {
    attributed: result.detected,
    confidence: result.confidence,
    modelFamily: result.modelFamily,
  };
}

Kubernetes v1.37 brings the Workload and PodGroup APIs, Workload-Aware Preemption, and shared DRA ResourceClaims to beta, all of which have direct implications for how multi-agent pipelines behave under resource pressure. The core value for observability is that gang scheduling enforces atomic start semantics — either the full agent cohort starts, or none of them do. This eliminates the partial-start failure mode where some agents have already emitted provenance records before the run was actually valid, which creates ghost entries in your audit log that have to be cleaned up manually. Source: [Kubernetes v1.37: Advancing Workload-Aware Scheduling] — Kubernetes.io (https://kubernetes.io/blog/2026/09/08/kubernetes-v1-37-advancing-workload-aware-scheduling/)

AIThe connection to per-phase provenance logging is architectural: your provenance middleware can treat a PodGroup start event as a run initialization signal, writing a single run-open record with all participating agent identifiers before any agent emits a phase record. This gives you a clean run boundary in the audit log and makes the sequence number space well-defined from the first step. When combined with WAP's atomic preemption, you get a cluster-level guarantee that your provenance log either contains a complete run or a cleanly closed incomplete run — never an ambiguous fragment.

// run-open record written atomically when PodGroup becomes ready
// triggered by a Kubernetes event watch on PodGroup status transition

interface RunOpenRecord {
  runId: string;
  podGroupName: string;
  participatingAgents: Array<{
    agentId: string;
    podName: string;
    nodeAssigned: string;
  }>;
  scheduledAt: number;
  pipelineVersion: string;
  status: 'open';
}

async function onPodGroupReady(
  event: { podGroupName: string; pods: Array<{ name: string; agentId: string; nodeName: string }> },
  provenanceStore: { insertRunOpen(record: RunOpenRecord): Promise<void> },
  pipelineVersion: string
): Promise<string> {
  const runId = crypto.randomUUID();

  const record: RunOpenRecord = {
    runId,
    podGroupName: event.podGroupName,
    participatingAgents: event.pods.map(p => ({
      agentId: p.agentId,
      podName: p.name,
      nodeAssigned: p.nodeName,
    })),
    scheduledAt: Date.now(),
    pipelineVersion,
    status: 'open',
  };

  await provenanceStore.insertRunOpen(record);
  return runId;  // propagate runId to all agents via env or config map
}

Lilian Weng's treatment of recursive self-improvement traces the concept from Good's ultraintelligent machine through to modern feedback loops where a model's outputs influence the cognitive machinery that produces future outputs. In production agentic systems, the practically relevant version of this is much narrower but still real: an agent that writes synthesized knowledge back to a retrieval store, which is then retrieved in a future session, is executing a feedback loop where the quality of prior outputs shapes future reasoning. The governance problem is that this loop is invisible unless you are specifically logging memory write events and connecting them to the session provenance envelope. Source: [Harness Engineering for Self-Improvement] — Lilian Weng (https://lilianweng.github.io/posts/2026-07-04-harness/)

AIThe engineering response is to treat every artifact that an agent can both read and write as a versioned, audited resource. Memory store entries, retrieved document indexes, and injected tool schemas should all carry monotonic version IDs, and every read and write of those resources should emit a provenance event. This gives you a complete picture of how the agent's behavioral context evolved over time — which is the only way to answer the question 'why did this agent start making different decisions in week three than it did in week one' without resorting to ad hoc log archaeology.

// Integrity check: detect drift in agent's dependency artifacts between sessions
import { createHash } from 'crypto';

interface ArtifactBaseline {
  artifactKey: string;
  baselineHash: string;
  establishedAt: number;
  establishedByRunId: string;
}

async function verifyArtifactIntegrity(
  artifactKey: string,
  currentContent: string,
  baseline: ArtifactBaseline,
  runId: string,
  onDriftDetected: (key: string, expected: string, actual: string, runId: string) => Promise<void>
): Promise<boolean> {
  const currentHash = createHash('sha256').update(currentContent).digest('hex');

  if (currentHash !== baseline.baselineHash) {
    await onDriftDetected(artifactKey, baseline.baselineHash, currentHash, runId);
    return false;  // caller decides whether to halt or continue with alert
  }

  return true;
}

// Call before each session start for critical artifacts
// onDriftDetected should log to provenance store AND optionally trigger human review