Critic Agents That Actually Catch Drift: Designing Feedback Loops for High-Stakes Agentic Systems in the First Six Weeks

Google's Biomarker Discovery Framework uses a multi-agent architecture where hypothesis generation, statistical analysis, adversarial validation, and literature-grounded reasoning each occupy distinct roles — and the adversarial agent has the authority to block progression, not just annotate concern. Across 9,279 participant-observations in three cohorts, this structure successfully recovered known clinical signals while preventing the spurious correlation artifacts that plague single-agent pipelines operating on sensor data. Source: [An AI tool for prioritizing candidate biomarkers from wearable sensor data] — Google Research (https://research.google/blog/an-ai-tool-for-prioritizing-candidate-biomarkers-from-wearable-sensor-data/)

AIThe pattern generalizes directly to any high-stakes decision pipeline: the critic agent needs teeth, not a voice. Placing it as a blocking node between the generator and the consumer — with a well-defined rejection schema that writes structured evidence to your observability store — gives you both a governance artifact and a drift signal you can trend over time. Without the rejection schema, you have a critic that runs but cannot be audited.

// TypeScript — Critic node in a LangGraph supervisor graph
// Critic returns a structured verdict, not a string

interface CriticVerdict {
  approved: boolean;
  confidence: number;       // 0.0 – 1.0
  drift_flags: string[];    // named deviation categories
  evidence_summary: string; // goes to observability store
  requires_human: boolean;
}

async function criticNode(
  state: AgentState
): Promise<Partial<AgentState>> {
  const verdict: CriticVerdict = await runCriticLLM({
    candidate_output: state.last_agent_output,
    baseline_distribution: state.baseline_stats,
    rejection_schema: CRITIC_SCHEMA,
  });

  await driftStore.record({
    run_id: state.run_id,
    phase: state.current_phase,
    verdict,
    timestamp: new Date().toISOString(),
  });

  return {
    critic_verdict: verdict,
    next_node: verdict.approved
      ? verdict.requires_human ? 'human_review' : 'downstream'
      : 'rejection_handler',
  };
}

The production LangGraph booking agent case study is instructive precisely because the motivation for Postgres wasn't observability — it was multi-interface state sharing. But the architectural consequence is that every checkpoint written to Postgres is also a timestamped, queryable audit record of agent decisions across the full session graph. Source: [Building a Proper Backend for My LangGraph AI Agent] — Towards Data Science (https://towardsdatascience.com/building-a-proper-backend-for-my-langgraph-ai-agent/)

AIFor drift detection specifically, the pattern you want is a critic agent that reads from the same checkpoint store on a scheduled basis — not just during live sessions — and computes rolling statistics over output distributions, tool call frequencies, and state transition patterns. This is the observability move that turns a durable backend from a reliability feature into a governance instrument. Connect this to your drift trend store from block_1 and you have a closed audit loop.

// TypeScript — Scheduled drift monitor querying Postgres checkpoints
// Runs every 6 hours during the first 6 weeks post-deployment

const DRIFT_WINDOW_DAYS = 7;
const DRIFT_THRESHOLD = 0.15; // KL divergence threshold

async function runDriftMonitor(pool: Pool): Promise<void> {
  const recent = await pool.query<CheckpointRow>(`
    SELECT run_id, phase, output_vector, created_at
    FROM agent_checkpoints
    WHERE created_at > NOW() - INTERVAL '${DRIFT_WINDOW_DAYS} days'
    ORDER BY created_at ASC
  `);

  const baseline = await loadBaselineDistribution(); // week-1 snapshot
  const current = computeOutputDistribution(recent.rows);
  const driftScore = klDivergence(baseline, current);

  await metricsStore.record({ driftScore, window: DRIFT_WINDOW_DAYS });

  if (driftScore > DRIFT_THRESHOLD) {
    await humanReviewQueue.enqueue({
      severity: 'HIGH',
      drift_score: driftScore,
      sample_run_ids: recent.rows.slice(-5).map(r => r.run_id),
    });
  }
}

ADOP's reference architecture on AWS Bedrock decomposes data engineering work into specialized agents that each own a discrete phase of the Bronze-to-Gold transformation, with configurable controls designed to support compliance requirements at each handoff. The structural insight is that governance is not a layer on top of the pipeline — it is embedded in the handoff protocol between agents. Source: [Agentic Data Operations Platform (ADOP): Data engineering into hours] — AWS Machine Learning Blog (https://aws.amazon.com/blogs/machine-learning/agentic-data-operations-platform-adop-data-engineering-into-hours/)

AIThis maps cleanly onto the critic loop design for decision pipelines: if you model each decision phase as a named lifecycle stage with a defined output schema and a critic that validates that schema before the next stage begins, you get per-phase provenance for free. The drift monitoring problem becomes tractable because you can isolate which phase's output distribution shifted, rather than trying to diagnose a diffuse change in end-to-end behavior. Combined with durable Postgres checkpoints from block_2, each phase transition becomes an auditable record.

// TypeScript — Phase gate enforcing output schema + critic verdict before handoff

const PhaseOutputSchema = z.object({
  phase: z.enum(['bronze', 'silver', 'gold']),
  output_payload: z.record(z.unknown()),
  schema_version: z.string(),
  critic_verdict: z.object({
    approved: z.boolean(),
    drift_flags: z.array(z.string()),
    confidence: z.number().min(0).max(1),
  }),
});

async function phaseGate(
  phaseOutput: unknown,
  phase: 'bronze' | 'silver' | 'gold',
  provenanceStore: ProvenanceStore
): Promise<'proceed' | 'reject' | 'human_review'> {
  const parsed = PhaseOutputSchema.safeParse(phaseOutput);
  if (!parsed.success) {
    await provenanceStore.logRejection({ phase, reason: 'schema_invalid' });
    return 'reject';
  }
  const { critic_verdict } = parsed.data;
  await provenanceStore.logTransition({ phase, critic_verdict });
  if (!critic_verdict.approved) return 'reject';
  if (critic_verdict.confidence < 0.7) return 'human_review';
  return 'proceed';
}

Chip Huyen's pitfall taxonomy is most useful for critic design because it identifies where variance enters the pipeline — and a critic agent can only be effective if its scope matches the failure mode it is targeting. Deploying a semantic correctness critic against a pipeline that is primarily failing on schema violations produces false confidence, not drift control. Source: [Common pitfalls when building generative AI applications] — Chip Huyen (https://huyenchip.com//2025/01/16/ai-engineering-pitfalls.html)

AIThe six-week window is when these mismatches surface hardest: your baseline was established on a narrow evaluation set, your critic was scoped too broadly or too narrowly, and real-world input distribution shifts expose both gaps simultaneously. The architectural move is to instrument each critic with its own rejection category taxonomy from day one — not as a logging nicety but as the data structure that lets you correlate which critic is firing, on which input types, with what frequency, so you can adjust critic scope dynamically rather than waiting for an incident to reveal the mismatch.

// TypeScript — Decomposed critic registry with per-category rejection telemetry

type CriticCategory = 'schema' | 'semantic' | 'business_rule';

interface CriticResult {
  category: CriticCategory;
  passed: boolean;
  rejection_code?: string;
  detail?: string;
}

async function runDecomposedCritics(
  output: AgentOutput,
  context: EvaluationContext
): Promise<CriticResult[]> {
  const results = await Promise.all([
    schemaCritic.evaluate(output),
    semanticCritic.evaluate(output, context),
    businessRuleCritic.evaluate(output, context),
  ]);

  await telemetry.recordCriticRun({
    run_id: context.run_id,
    results,
    timestamp: new Date().toISOString(),
    input_type: context.input_classification,
  });

  return results;
}

// Drift alert if any category rejection rate exceeds 10% in rolling 24h
async function checkCategoryDriftRate(
  category: CriticCategory,
  pool: Pool
): Promise<number> {
  const row = await pool.query<{ rejection_rate: number }>(`
    SELECT COUNT(*) FILTER (WHERE passed = false)::float /
           NULLIF(COUNT(*), 0) AS rejection_rate
    FROM critic_telemetry
    WHERE category = $1
      AND timestamp > NOW() - INTERVAL '24 hours'
  `, [category]);
  return row.rows[0]?.rejection_rate ?? 0;
}

AIThe simulation tradeoff the Latent Space framing surfaces — lower fidelity, dramatically lower cost, dramatically higher velocity — applies directly to the critic baseline problem. You don't need production traffic to establish a baseline; you need a representative sample of inputs run through your agent under controlled conditions, with outputs labeled and stored as the reference distribution. This is exactly the kind of pre-deployment work that determines whether your six-week window is a governed rollout or an extended debugging session.

The practical implementation is a pre-deployment simulation harness that runs your agent against a synthetic input corpus, captures critic verdict distributions by category, and writes those distributions to your drift store as the versioned baseline. When production traffic arrives, every critic evaluation is compared against this baseline rather than against an implicit expectation in someone's head. Source: [AINews: 10% worse, 100x cheaper, 10000x faster: Why Simulation is taking over] — Latent Space (https://www.latent.space/p/ainews-10-worse-100x-cheaper-10000x)

// TypeScript — Pre-deployment baseline builder using synthetic corpus

async function buildBaselineFromSimulation(
  agent: AgentRunner,
  syntheticCorpus: AgentInput[],
  baselineVersion: string,
  driftStore: DriftStore
): Promise<void> {
  const verdicts: CriticResult[][] = [];

  for (const input of syntheticCorpus) {
    const output = await agent.run(input);
    const results = await runDecomposedCritics(output, {
      run_id: `baseline-${baselineVersion}-${crypto.randomUUID()}`,
      input_classification: input.type,
    });
    verdicts.push(results);
  }

  const distribution = computeCriticDistribution(verdicts);

  await driftStore.saveBaseline({
    version: baselineVersion,
    distribution,
    corpus_size: syntheticCorpus.length,
    created_at: new Date().toISOString(),
    is_simulation: true,
  });

  console.log(`Baseline v${baselineVersion} written — ${syntheticCorpus.length} simulated runs`);
}