Alerting for Things You Can't Directly Measure: Detecting Hallucination and Pipeline Drift in Production Agentic Systems

The core insight from the Tickloom case study is that a DSL isn't just a developer ergonomics choice — it's an observability contract. When your LLM generates within a formal grammar, every output is either valid or invalid by construction, and you can instrument that boundary with hard metrics rather than soft heuristics. Your alert threshold becomes 'parse failure rate exceeded 2% over a 5-minute window' rather than 'cosine similarity to expected output dropped below 0.8,' which is infinitely more actionable for an on-call engineer. Source: [DSLs Enable Reliable Use of LLMs] — Unmesh Joshi (https://martinfowler.com/articles/llm-and-dsls.html)

From an SRE perspective, this pattern also gives you a clean separation between syntax-layer alerts (the DSL rejected the output) and semantic-layer alerts (the output parsed fine but downstream tool calls returned anomalous results). Keeping these two alert classes separate in your runbooks matters because the remediation paths are completely different: a syntax failure often points to model version drift or prompt regression, while a semantic failure points to tool behavior change or data distribution shift. [AI Synthesis] Pairing DSL validation with structured trace spans — one span per generation attempt, tagged with parse_status and schema_version — lets you build dashboards that separate these failure modes without custom log parsing.

// TypeScript: DSL validation span instrumentation for agentic pipeline
import { trace, SpanStatusCode } from '@opentelemetry/api';

const DSL_SCHEMA_VERSION = 'v2.3.1';

async function validateAgentOutput(
  rawOutput: string,
  dslParser: (input: string) => ParseResult
): Promise<ValidatedOutput> {
  const tracer = trace.getTracer('agent-pipeline');
  const span = tracer.startSpan('dsl.validation');

  try {
    const result = dslParser(rawOutput);

    span.setAttributes({
      'dsl.schema_version': DSL_SCHEMA_VERSION,
      'dsl.parse_status': result.valid ? 'ok' : 'failed',
      'dsl.failure_reason': result.error ?? '',
      'dsl.output_token_count': rawOutput.length,
    });

    if (!result.valid) {
      span.setStatus({ code: SpanStatusCode.ERROR, message: result.error });
      // Metric counter — wire this to your alerting rule
      parseFailureCounter.add(1, {
        schema_version: DSL_SCHEMA_VERSION,
        failure_type: result.errorCategory ?? 'unknown',
      });
      throw new DslValidationError(result.error);
    }

    return result.parsed;
  } finally {
    span.end();
  }
}

Addy Osmani's framing of 'light factory' (humans reading and approving) versus 'dark factory' (agents scoping, building, and shipping without review) maps cleanly onto a supervision topology problem in multi-agent systems. In a dark factory configuration, your alerting infrastructure isn't supplementing human oversight — it's replacing it entirely, which means your false-negative rate on hallucination detection becomes a direct proxy for your system's safety margin. Source: [Software Factories, Light and Dark] — Addy Osmani (https://addyosmani.com/blog/software-factories/)

AIA practical pattern here is to assign each agent node an autonomy_tier value (0 = always escalate, 1 = escalate on anomaly, 2 = self-route with logging, 3 = fire and forget) and encode that in your OpenTelemetry span attributes. Your alerting rules then become tier-aware: a tier-3 node that starts producing outputs with low confidence scores should automatically trigger a tier reclassification review, not just a one-off alert. This keeps your human review queue focused on structural governance decisions rather than individual output triage.

// TypeScript: Autonomy tier routing with span instrumentation
type AutonomyTier = 0 | 1 | 2 | 3;

interface AgentNode {
  nodeId: string;
  autonomyTier: AutonomyTier;
  confidenceThreshold: number;
}

async function routeByAutonomyTier(
  node: AgentNode,
  output: AgentOutput,
  span: Span
): Promise<RoutingDecision> {
  span.setAttributes({
    'agent.node_id': node.nodeId,
    'agent.autonomy_tier': node.autonomyTier,
    'agent.output_confidence': output.confidence,
  });

  if (node.autonomyTier === 0) {
    return { action: 'escalate', reason: 'tier-0 always escalates' };
  }

  if (node.autonomyTier === 1 && output.confidence < node.confidenceThreshold) {
    anomalyCounter.add(1, { node_id: node.nodeId, tier: '1' });
    return { action: 'escalate', reason: 'confidence below threshold' };
  }

  if (output.confidence < node.confidenceThreshold * 0.6) {
    // Significant confidence drop — flag tier reclassification
    tierReclassificationGauge.record(1, {
      node_id: node.nodeId,
      current_tier: String(node.autonomyTier),
    });
  }

  return { action: 'continue', tier: node.autonomyTier };
}

Simon Willison's observation about the ROI shift in reverse-engineering work applies directly to observability tooling for agentic pipelines. The cost of building a custom behavioral baseline collector — one that records tool call sequences, argument distributions, and output token patterns per agent node — has dropped from 'requires a dedicated sprint' to 'afternoon task with a coding agent.' The more important point is his warning about unstable APIs: LLM providers are exactly like undocumented device firmware. Your pipeline's behavioral baseline can shift because the model changed, because the sampling parameters drifted, or because a system prompt got truncated by a context window change — and none of those show up as HTTP errors. Source: [Reverse-engineering is cheap now] — Simon Willison (https://simonwillison.net/2026/Jul/20/cheap-reverse-engineering/#atom-everything)

AIA practical alert strategy here is to maintain a rolling behavioral fingerprint per agent node: hash the ordered sequence of tool names called per pipeline execution, bucket them by input category, and alert when the distribution shifts beyond a configured threshold over a 1-hour window. This is analogous to a p99 latency alert, but for behavior rather than performance — and it catches the class of hallucination where the model produces a plausible output via a completely wrong reasoning path, which is the failure mode that most output-only validators miss entirely.

// TypeScript: Tool-call sequence fingerprinting for behavioral drift detection
import { createHash } from 'crypto';

interface ToolCallRecord {
  toolName: string;
  nodeId: string;
  argHash: string; // hash of sanitized args, not raw values
  timestamp: number;
}

function fingerprintExecution(calls: ToolCallRecord[]): string {
  const sequence = calls
    .sort((a, b) => a.timestamp - b.timestamp)
    .map(c => `${c.nodeId}:${c.toolName}`)
    .join('|');
  return createHash('sha256').update(sequence).digest('hex').slice(0, 16);
}

async function checkBehavioralDrift(
  nodeId: string,
  currentFingerprint: string,
  baselineStore: FingerprintStore,
  driftThreshold = 0.15 // 15% distribution shift triggers alert
): Promise<DriftCheckResult> {
  const recentFingerprints = await baselineStore.getWindow(nodeId, '1h');
  const matchRate = recentFingerprints.filter(
    fp => fp === currentFingerprint
  ).length / recentFingerprints.length;

  const driftScore = 1 - matchRate;

  behavioralDriftGauge.record(driftScore, { node_id: nodeId });

  if (driftScore > driftThreshold) {
    return { drifted: true, driftScore, nodeId };
  }

  return { drifted: false, driftScore, nodeId };
}

The Kubeflow/Headlamp integration exposes something genuinely useful for agentic platform teams: when every pipeline stage is a CRD, your Kubernetes event stream becomes a structured audit log for pipeline behavior. A PipelineRun resource that transitions from Running to Failed carries a reason field that you can parse and route to incident management, and the event history gives you a timeline that's automatically correlated with pod scheduling, resource pressure, and node events — context that raw LLM output logs don't carry. Source: [Operating AI/ML Workloads on Kubernetes: A Headlamp Plugin for Kubeflow] — (https://kubernetes.io/blog/2026/07/13/introducing-headlamp-plugin-for-kubeflow/)

AIFor agentic pipelines specifically, the pattern worth adopting is to model each agent execution as a CRD with a status.conditions array that tracks not just Kubernetes-level health but semantic pipeline health: a condition like SemanticValidationPassed or BehavioralBaselineMatched embedded in the resource status gives your alerting stack a single source of truth that aggregates both infrastructure and application-layer signals. Prometheus can scrape CRD status conditions via kube-state-metrics custom resource configuration, which means you can write a single alerting rule that fires when any agent node has had SemanticValidationPassed=False for more than two consecutive executions — no custom exporter required.

# Prometheus alerting rule for CRD-level semantic validation failures
# Deploy via PrometheusRule CRD (kube-prometheus-stack)
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: agent-pipeline-semantic-alerts
  namespace: kubeflow
spec:
  groups:
    - name: agent.semantic.validation
      interval: 30s
      rules:
        - alert: AgentSemanticValidationFailing
          expr: |
            kube_customresource_status_condition{
              customresource_kind="AgentPipelineRun",
              condition="SemanticValidationPassed",
              status="false"
            } == 1
          for: 2m
          labels:
            severity: warning
            team: platform-ai
          annotations:
            summary: "Agent node {{ $labels.node_id }} failing semantic validation"
            description: >-
              Pipeline run {{ $labels.name }} has had SemanticValidationPassed=False
              for 2+ consecutive executions. Possible hallucination or DSL schema drift.
            runbook_url: "https://wiki.internal/runbooks/agent-semantic-validation"

        - alert: AgentBehavioralDriftDetected
          expr: |
            agent_behavioral_drift_score > 0.15
          for: 5m
          labels:
            severity: critical
            team: platform-ai
          annotations:
            summary: "Behavioral drift on agent node {{ $labels.node_id }}"
            description: >-
              Tool-call sequence distribution has shifted >15% over the last hour.
              Possible model version change or prompt regression.

Grafana's observation about frontend observability applies with force to agentic pipelines: you can have a p99 latency of 800ms, a 0.1% error rate, and a perfectly healthy Kubernetes cluster while your agent is confidently producing wrong answers for 20% of requests in a specific intent category. The gap between infrastructure health and user outcome is wider in agentic systems than in conventional web apps because the most common failure mode — a plausible but wrong output — is invisible to every infrastructure-layer metric. Source: [Beyond performance monitoring: Understand the user experience with Grafana Cloud Frontend Observability] — (https://grafana.com/blog/beyond-performance-monitoring-understand-the-user-experience-with-grafana-cloud-frontend-observability/)

AIThe alert architecture implication is that you need at least two distinct metric layers: a pipeline health layer (latency, parse failure rate, tool call error rate, token budget exhaustion) and a task outcome layer (intent completion rate, downstream correction rate, user retry rate, explicit feedback signals). The task outcome layer is harder to instrument because it requires you to define done, but it's the only layer that can catch the hallucination pattern where the model produces a coherent, well-parsed, on-time output that is factually wrong. Connecting these two layers — correlating a spike in user retry rate with a concurrent change in behavioral fingerprint distribution — is where your actual root cause analysis starts.

// TypeScript: Dual-layer metric emission for pipeline health + task outcomes
import { metrics } from '@opentelemetry/api';

const meter = metrics.getMeter('agent-pipeline-outcomes');

// Pipeline health layer
const parseFailureRate = meter.createCounter('agent.dsl.parse_failures');
const toolCallErrors = meter.createCounter('agent.tool.errors');
const pipelineLatency = meter.createHistogram('agent.pipeline.duration_ms');

// Task outcome layer
const intentCompletionRate = meter.createCounter('agent.task.completions', {
  description: 'Completed tasks where downstream validation confirmed intent was met',
});
const userRetrySignal = meter.createCounter('agent.task.user_retries', {
  description: 'User re-submitted same intent within 60s — proxy for dissatisfaction',
});

function emitPipelineOutcome(
  executionResult: PipelineExecutionResult
): void {
  const attrs = {
    node_id: executionResult.nodeId,
    intent_category: executionResult.intentCategory,
    model_version: executionResult.modelVersion,
    autonomy_tier: String(executionResult.autonomyTier),
  };

  pipelineLatency.record(executionResult.durationMs, attrs);

  if (executionResult.dslParseFailed) {
    parseFailureRate.add(1, attrs);
  }

  if (executionResult.taskCompleted) {
    intentCompletionRate.add(1, attrs);
  }

  if (executionResult.userRetried) {
    userRetrySignal.add(1, attrs);
  }
}

Lilian Weng's treatment of scaling laws as a framework for predicting the relationship between compute, loss, and capability has a direct operational implication for agentic platform teams: model behavior is not arbitrary, it shifts in predictable directions as scale increases. A model with lower training loss will be more capable at multi-step reasoning, more likely to attempt tool use without being prompted, and more likely to generate plausible-but-wrong outputs in domains that are underrepresented in training data — all of which affect your pipeline's behavioral fingerprint. Source: [Scaling Laws, Carefully] — Lilian Weng (https://lilianweng.github.io/posts/2026-06-24-scaling-laws/)

AIThe alert strategy implication is to treat LLM provider release announcements as change events in your incident management system. When a provider announces a model update, you should automatically trigger a baseline re-calibration job that runs your behavioral fingerprinting suite against the new model version in a shadow environment, computes the expected distribution shift, and updates your drift alert thresholds accordingly — rather than letting your existing thresholds fire spuriously for a week post-rollout. This is the same discipline you'd apply to a database engine upgrade or a dependency major version bump: planned, measured, with observable rollback criteria.

// TypeScript: Model version change event handler with baseline recalibration
interface ModelVersionChangeEvent {
  provider: string;
  previousVersion: string;
  newVersion: string;
  announcedAt: Date;
  changeType: 'minor' | 'major' | 'fine-tune-base-update';
}

async function handleModelVersionChange(
  event: ModelVersionChangeEvent,
  fingerprintStore: FingerprintStore,
  shadowRunner: ShadowPipelineRunner
): Promise<RecalibrationResult> {
  // Run behavioral fingerprint suite in shadow env against new model version
  const shadowResults = await shadowRunner.runFingerprints({
    modelVersion: event.newVersion,
    testCases: await fingerprintStore.getCanonicalTestCases(),
    sampleSize: event.changeType === 'major' ? 500 : 100,
  });

  const previousBaseline = await fingerprintStore.getBaseline(event.previousVersion);
  const distributionDelta = computeDistributionDelta(shadowResults, previousBaseline);

  if (distributionDelta > 0.3 && event.changeType === 'minor') {
    // Unexpected large shift for a minor version — escalate before rollout
    await escalate({
      severity: 'warning',
      message: `Unexpected behavioral delta ${distributionDelta} for minor model update`,
      event,
      distributionDelta,
    });
  }

  // Update alert thresholds to account for expected new baseline
  await fingerprintStore.updateBaselineThresholds({
    modelVersion: event.newVersion,
    expectedDrift: distributionDelta * 0.5, // allow 50% of observed delta as normal variation
    effectiveFrom: event.announcedAt,
  });

  return { recalibrated: true, distributionDelta, newVersion: event.newVersion };
}