Supervisor Topologies for Cascading Failure Detection: Building Self-Healing Agentic Pipelines Without Manual Intervention

The architectural instinct for most platform teams is to build one supervisor node and route all agent health signals through it. This works until it doesn't — and it fails precisely when you need it most, during a cascade, because the supervisor's own context window and decision latency become the bottleneck. The better model is a tiered hierarchy where each supervisor owns a coherent slice: a data-ingestion supervisor, a validation supervisor, a reporting supervisor. Each one knows its agents deeply and emits a normalized failure envelope upward rather than raw stack traces. The parent supervisor sees clean, typed signals and can apply policy — quarantine, retry with model swap, page a human — without drowning in noise.

The fintech compliance case (cutting a two-day manual reporting process to two hours) illustrates exactly why domain scoping matters: aggregating data from multiple sources, validating it, and generating a compliant report are three distinct failure domains with different remediation strategies. A failure in data ingestion shouldn't block the validation supervisor from reporting that the previous run's validated data is still clean. Keeping those domains isolated is what lets the system degrade gracefully rather than halt entirely. Source: [How a Leading Fintech Cuts Weekly Compliance Reporting from 2 Days to 2 Hours] — blog.crewai.com (https://blog.crewai.com/how-a-leading-fintech-cuts-weekly-compliance-reporting-from-2-days-to-2-hours/)

// TypeScript: Typed failure envelope for inter-supervisor escalation
type FailureSeverity = 'recoverable' | 'degraded' | 'critical';

interface AgentFailureEnvelope {
  agentId: string;
  domainId: string;
  severity: FailureSeverity;
  errorCode: string;
  retryCount: number;
  lastKnownGoodStateRef: string | null; // pointer to provenance log entry
  timestamp: string;
}

interface SupervisorEscalation {
  supervisorId: string;
  aggregatedSeverity: FailureSeverity;
  affectedAgentCount: number;
  rootCauseHypothesis: string;
  recommendedAction: 'retry' | 'reroute' | 'quarantine' | 'page_human';
  failureEnvelopes: AgentFailureEnvelope[];
}

async function evaluateAndEscalate(
  envelopes: AgentFailureEnvelope[],
  supervisorId: string
): Promise<SupervisorEscalation | null> {
  const critical = envelopes.filter(e => e.severity === 'critical');
  if (critical.length === 0) return null;

  const cascading = critical.length > 1;
  return {
    supervisorId,
    aggregatedSeverity: 'critical',
    affectedAgentCount: critical.length,
    rootCauseHypothesis: cascading ? 'shared-dependency failure' : critical[0].errorCode,
    recommendedAction: cascading ? 'quarantine' : 'retry',
    failureEnvelopes: critical,
  };
}

The emerging discipline the Thoughtworks report calls 'harness engineering' is exactly what critic agent design is at the agentic layer — the scaffolding that determines whether your agent's output is trustworthy before it propagates downstream. This isn't just about catching bugs; it's about giving the supervisor a typed, queryable signal rather than a blob of text. When a critic returns a verdict like `{ coherence: 0.82, compliance: 0.61, recommendation: 'reroute' }`, the supervisor can apply deterministic policy: below a compliance threshold, quarantine and page a human; below coherence but above compliance, retry with a stronger model. That's a governance surface you can reason about and audit. Source: [Fragments: July 21] — martinfowler.com (https://martinfowler.com/fragments/2026-07-21.html)

Test-time compute research shows that giving a model more 'thinking time' via chain-of-thought or extended reasoning can substantially improve output quality on complex tasks — which means your critic agent itself can be configured to run extended reasoning when confidence is borderline, rather than forcing a hard binary verdict. This tiered critic strategy (fast cheap check → deep reasoning check on borderline cases) mirrors exactly how a senior engineer applies judgment: quick scan first, deep review only when something looks off. Source: [Why We Think] — lilianweng.github.io (https://lilianweng.github.io/posts/2025-05-01-thinking/)

// TypeScript: Critic verdict with tiered confidence routing
interface CriticVerdict {
  criticId: string;
  dimension: 'schema' | 'compliance' | 'coherence';
  score: number; // 0.0 - 1.0
  pass: boolean;
  reasoning: string | null; // populated only when score is borderline
}

interface SupervisorRoutingPolicy {
  complianceFloor: number;
  coherenceFloor: number;
  borderlineThreshold: number; // triggers extended reasoning in critic
}

function routeOnVerdicts(
  verdicts: CriticVerdict[],
  policy: SupervisorRoutingPolicy
): 'approve' | 'retry_stronger' | 'quarantine_human' {
  const compliance = verdicts.find(v => v.dimension === 'compliance');
  const coherence = verdicts.find(v => v.dimension === 'coherence');

  if (!compliance || compliance.score < policy.complianceFloor) {
    return 'quarantine_human';
  }
  if (!coherence || coherence.score < policy.coherenceFloor) {
    return 'retry_stronger';
  }
  return 'approve';
}

The fintech compliance pipeline case is the clearest production example of why per-phase provenance is non-negotiable for agentic systems under governance pressure. When you're aggregating data from multiple regulated sources and a critic catches a compliance violation, the question isn't just 'what failed' — it's 'which source introduced the bad data, at which transformation step, under which agent version.' Without immutable phase logs, you're doing forensics on a crime scene where the evidence has been partially overwritten. With them, the supervisor can pinpoint the blast radius in seconds and either replay from a clean checkpoint or escalate only the affected branch. Source: [How a Leading Fintech Cuts Weekly Compliance Reporting from 2 Days to 2 Hours] — blog.crewai.com (https://blog.crewai.com/how-a-leading-fintech-cuts-weekly-compliance-reporting-from-2-days-to-2-hours/)

Grafana Cloud's cost attribution model — where spend is tagged at the team, service, and project level rather than rolled into a single bill — is a useful structural analogy here. The same principle applies to failure attribution: tag every phase log entry with the supervisor domain, agent ID, and task version so that when you're debugging a cascade, you have a filterable, attributable record rather than a monolithic event stream. This also happens to make your observability spend more defensible — if your platform team can show that 80% of your tracing volume comes from the compliance agent domain, you can make a data-driven case for investing in more targeted instrumentation there. Source: [Cost attribution in Grafana Cloud] — grafana.com (https://grafana.com/blog/cost-attribution-in-grafana-cloud-manage-spend-across-observability-and-testing-workflows/)

// TypeScript: Immutable phase log entry for PostgreSQL
import { Pool } from 'pg';

interface PhaseLogEntry {
  phaseId: string;          // uuid
  supervisorDomain: string; // e.g. 'compliance', 'ingestion'
  agentId: string;
  agentVersion: string;
  taskId: string;
  inputRef: string;         // hash or s3 key of input snapshot
  outputRef: string;        // hash or s3 key of output snapshot
  verdictRef: string | null; // critic verdict ID if applicable
  status: 'success' | 'failed' | 'degraded';
  createdAt: string;        // ISO 8601, immutable after insert
}

async function writePhaseLog(pool: Pool, entry: PhaseLogEntry): Promise<void> {
  await pool.query(
    `INSERT INTO phase_logs
      (phase_id, supervisor_domain, agent_id, agent_version, task_id,
       input_ref, output_ref, verdict_ref, status, created_at)
     VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10)
     ON CONFLICT (phase_id) DO NOTHING`, // immutability enforced at DB level
    [
      entry.phaseId, entry.supervisorDomain, entry.agentId, entry.agentVersion,
      entry.taskId, entry.inputRef, entry.outputRef, entry.verdictRef,
      entry.status, entry.createdAt
    ]
  );
}

async function getLastCleanCheckpoint(pool: Pool, taskId: string, domain: string): Promise<PhaseLogEntry | null> {
  const { rows } = await pool.query(
    `SELECT * FROM phase_logs
     WHERE task_id = $1 AND supervisor_domain = $2 AND status = 'success'
     ORDER BY created_at DESC LIMIT 1`,
    [taskId, domain]
  );
  return rows[0] ?? null;
}

The smart model routing trend — companies dynamically selecting models based on task complexity, cost, and latency — is being picked up primarily as a cost play, but the more interesting application for agentic platform teams is as a supervisor-invoked remediation action. When your compliance critic returns a low-confidence verdict and the primary agent is already at its retry limit, the supervisor has three options: escalate to a human, quarantine the task, or swap in a different model with a different failure mode. That third option is under-explored and under-tooled. A routing table that maps `(domain, failure_type, retry_count)` to a ranked model list gives the supervisor a concrete, auditable decision it can make without human input. Source: [The Pulse: a new trend, smart model routing] — blog.pragmaticengineer.com (https://blog.pragmaticengineer.com/the-pulse-a-new-trend-smart-model-routing/)

AIThe napkin math discipline — using back-of-envelope capacity reasoning to understand system limits before you hit them — applies directly here. If you know your primary reasoning model has a P99 latency of 8 seconds and your pipeline SLA is 30 seconds across five agents, you have exactly 6 seconds of slack before any single agent stall cascades into an SLA breach. That calculation tells you when the supervisor must reroute (not retry) and how much latency budget the fallback model is allowed to consume. Building those numbers into your supervisor's routing policy before you need them is the difference between a self-healing system and one that pages you at 2am.

// TypeScript: Supervisor model routing table keyed on failure state
type FailureState = {
  domain: string;
  retryCount: number;
  lastCriticScore: number;
  stallDurationMs: number;
};

type ModelRoute = {
  modelId: string;
  maxLatencyMs: number;
  rationale: string;
};

function resolveModelRoute(state: FailureState, slaRemainingMs: number): ModelRoute | 'quarantine_human' {
  if (state.lastCriticScore < 0.5) {
    return 'quarantine_human';
  }
  if (state.stallDurationMs > 5000 || slaRemainingMs < 8000) {
    return { modelId: 'gpt-4o-mini', maxLatencyMs: 3000, rationale: 'sla-pressure-reroute' };
  }
  if (state.retryCount >= 1 && state.lastCriticScore < 0.75) {
    return { modelId: 'o3', maxLatencyMs: 12000, rationale: 'quality-degraded-upscale' };
  }
  return { modelId: 'gpt-4o', maxLatencyMs: 8000, rationale: 'primary-model' };
}

The SymptomAI study is worth reading as an architecture case study, not just an AI health story. Across nearly 14,000 participants, the system conducted end-to-end symptom interviews and produced differential diagnoses — but the architecture positions clinician oversight as the validation layer above the agent's output, not as a gate the agent passes through. That's a deliberate topology choice: the agent handles the high-volume, high-repetition work of structured data collection and initial assessment; the human handles the high-stakes, low-volume decision of final diagnosis. Your platform team's compliance pipeline has the same structure — the agents handle data aggregation and normalization, and the human gate should sit at the point where a compliance decision with regulatory consequences is about to be written, not at every intermediate data transformation step. Source: [SymptomAI: Towards a conversational AI agent for everyday symptom assessment] — research.google (https://research.google/blog/symptomai-towards-a-conversational-ai-agent-for-everyday-symptom-assessment/)

The executive/engineer expectation gap surfaced at the Thoughtworks retreat is a useful reminder that human-in-the-loop design isn't just a technical decision — it's an organizational contract. If your supervisor escalates to a human reviewer without providing enough context for a real decision, that reviewer will either rubber-stamp or escalate further, and neither outcome is what your governance model is actually asking for. The digest that reaches the human gate should be actionable in isolation: what failed, what the supervisor already tried, what the reviewer is being asked to decide. That's a product design problem as much as an engineering one. Source: [Fragments: July 21] — martinfowler.com (https://martinfowler.com/fragments/2026-07-21.html)

// TypeScript: Human gate escalation digest — what the reviewer actually sees
interface EscalationDigest {
  escalationId: string;
  domain: string;
  severity: 'degraded' | 'critical';
  summary: string;            // plain-language: what failed and why
  affectedTaskIds: string[];
  supervisorActionsAttempted: string[]; // e.g. ['retry_primary', 'reroute_to_o3']
  lastCleanCheckpointRef: string;
  criticsScores: Record<string, number>; // e.g. { compliance: 0.43, coherence: 0.91 }
  recommendedAction: 'approve_replay' | 'reject_quarantine' | 'needs_human_judgment';
  expiresAt: string;          // ISO 8601 — auto-quarantine if no response
}

async function createEscalationDigest(
  supervisorEscalation: SupervisorEscalation,
  checkpointRef: string
): Promise<EscalationDigest> {
  return {
    escalationId: crypto.randomUUID(),
    domain: supervisorEscalation.supervisorId,
    severity: supervisorEscalation.aggregatedSeverity,
    summary: `${supervisorEscalation.affectedAgentCount} agents in ${supervisorEscalation.supervisorId} domain failed after ${supervisorEscalation.failureEnvelopes[0].retryCount} retries. Hypothesis: ${supervisorEscalation.rootCauseHypothesis}.`,
    affectedTaskIds: supervisorEscalation.failureEnvelopes.map(e => e.agentId),
    supervisorActionsAttempted: ['retry_primary', `reroute_${supervisorEscalation.recommendedAction}`],
    lastCleanCheckpointRef: checkpointRef,
    criticsScores: {}, // populated from critic verdicts
    recommendedAction: supervisorEscalation.aggregatedSeverity === 'critical' ? 'needs_human_judgment' : 'approve_replay',
    expiresAt: new Date(Date.now() + 30 * 60 * 1000).toISOString(),
  };
}

Simon Eskildsen's 'napkin math' approach — working through the fundamental physical and computational limits of a system with back-of-envelope calculations before optimizing — is exactly the discipline agentic platform architects are missing. Most supervisor designs are built by intuition ('three retries seems reasonable') rather than by deriving the numbers from first principles. If your compliance pipeline has a 30-second SLA and five sequential agent stages, and each stage has a 10% failure rate (optimistic for LLM calls under load), your expected pipeline success rate is 59%. That means four in ten runs will hit your supervisor's failure path — not an edge case, a routine occurrence that your supervisor topology needs to handle as the normal case, not the exception. Source: [Pushing software engineering limits with napkin math] — newsletter.pragmaticengineer.com (https://newsletter.pragmaticengineer.com/p/pushing-software-engineering-limits)

AIThe connection between napkin math and model selection is direct: smaller, faster models like the ones in Laguna S 2.1's class (competitive quality at a fraction of the cost and latency) change the failure recovery math significantly. If your fallback model is 10x cheaper and 3x faster than your primary, the cost of a supervisor-triggered reroute on every borderline output drops from a budget concern to a rounding error — which means you can set your critic score thresholds more aggressively and catch more marginal failures without blowing your inference budget. That's a number worth putting on a napkin before you finalize your routing policy.

// TypeScript: Cascade probability and retry budget calculator
function cascadeProbability(agentCount: number, perAgentFailureRate: number): number {
  return 1 - Math.pow(1 - perAgentFailureRate, agentCount);
}

function maxRetryBudget(params: {
  slaMs: number;
  agentCount: number;
  primaryModelLatencyMs: number;
  fallbackModelLatencyMs: number;
}): { primaryRetries: number; fallbackRetries: number } {
  const { slaMs, agentCount, primaryModelLatencyMs, fallbackModelLatencyMs } = params;
  const nominalExecutionMs = agentCount * primaryModelLatencyMs;
  const slackMs = slaMs - nominalExecutionMs;

  if (slackMs <= 0) return { primaryRetries: 0, fallbackRetries: 0 };

  // Allocate 60% of slack to primary retries, 40% to fallback
  const primaryRetries = Math.floor((slackMs * 0.6) / primaryModelLatencyMs);
  const fallbackRetries = Math.floor((slackMs * 0.4) / fallbackModelLatencyMs);

  return { primaryRetries, fallbackRetries };
}

// Example: 5-agent pipeline, 60s SLA, 8s primary, 2.5s fallback
const cascadeRisk = cascadeProbability(5, 0.10); // 0.41 — 41% of runs hit supervisor
const retryBudget = maxRetryBudget({
  slaMs: 60_000,
  agentCount: 5,
  primaryModelLatencyMs: 8_000,
  fallbackModelLatencyMs: 2_500,
});
// => { primaryRetries: 1, fallbackRetries: 3 } across entire pipeline
console.log({ cascadeRisk, retryBudget });