Decomposed Observability for LLM Pipelines: Catching Cascading Failures Before They Compound

The argument for removing direct database credentials from agents isn't purely security theater — it's also an observability architecture decision. When an agent calls a database directly, your only visibility is whatever query logging the database itself provides, which gives you SQL text but not agent context: which workflow triggered this, which supervisor node dispatched the agent, what the upstream reasoning state was. A mediation layer — a lightweight API service that wraps data access — lets you attach that context at the call site before forwarding the query, producing a trace that spans from supervisor intent to actual data retrieved. That's the foundation of decomposed observability: you can't decompose what you can't attribute. Source: [Stop giving your agents database credentials] — CrewAI (https://blog.crewai.com/stop-giving-your-agents-database-credentials/)

AIThe pattern here connects directly to what platform teams already do with service meshes and API gateways for microservices — the insight is that agentic tool calls deserve the same instrumentation discipline as service-to-service HTTP calls. A tool call that silently returns stale data or a malformed result propagates that corruption downstream through every dependent reasoning step, compounding the error in ways that are very hard to unwind in post-incident analysis. Treating tool boundaries as structured telemetry surfaces — with input/output hashing, latency histograms, and error classification — gives you the decomposed signal needed to isolate which step in a multi-agent chain introduced the failure.

// TypeScript: Tool call wrapper that emits structured telemetry
import { SpanStatusCode, trace } from '@opentelemetry/api';

type ToolCallMeta = {
  agentId: string;
  workflowId: string;
  toolName: string;
};

async function instrumentedToolCall<T>(
  meta: ToolCallMeta,
  fn: () => Promise<T>
): Promise<T> {
  const tracer = trace.getTracer('agent-tool-boundary');
  const span = tracer.startSpan(`tool.${meta.toolName}`);

  span.setAttributes({
    'agent.id': meta.agentId,
    'workflow.id': meta.workflowId,
    'tool.name': meta.toolName,
  });

  try {
    const result = await fn();
    span.setStatus({ code: SpanStatusCode.OK });
    return result;
  } catch (err) {
    span.setStatus({ code: SpanStatusCode.ERROR, message: String(err) });
    span.recordException(err as Error);
    throw err;
  } finally {
    span.end();
  }
}

The Strands Agents / Bedrock social intelligence architecture is a useful concrete case: you have a Reddit monitor agent, a Hacker News agent, a GitHub signals agent, and a Stack Overflow agent all feeding a correlation engine. Each of those agents has its own failure modes — rate limiting, stale cache, API schema drift, model miscalibration. Without per-agent health telemetry emitted as structured events, the correlation engine sees four inputs and has no way to distinguish 'GitHub stars crossed 2,400' from 'GitHub API returned a cached response from six hours ago that coincidentally shows 2,400 stars.' The downstream buying-intent signal looks identical. Source: [Multi-agent social intelligence with Strands Agents and Amazon Bedrock] — AWS Machine Learning Blog (https://aws.amazon.com/blogs/machine-learning/multi-agent-social-intelligence-with-strands-agents-and-amazon-bedrock/)

AIThis is the core challenge of decomposed observability in fan-in architectures: the aggregation step is where individual agent failures become invisible. The fix is to treat each agent's output as a typed payload that includes provenance metadata — source freshness timestamp, retrieval confidence, error flags — not just the semantic content. A supervisor node that receives a structured envelope can route low-confidence inputs to a verification step or a human-in-the-loop gate rather than silently folding them into the final answer. This pattern connects directly to the draft-then-verify trend: you're not just verifying the final output, you're verifying each input before aggregation.

// TypeScript: Typed agent output envelope with provenance metadata
type AgentOutputEnvelope<T> = {
  agentId: string;
  sourceTag: string;
  payload: T;
  freshness: {
    retrievedAt: string; // ISO8601
    maxAgeSeconds: number;
    isStale: boolean;
  };
  confidence: {
    score: number; // 0.0 - 1.0
    basis: 'retrieval' | 'model' | 'hybrid';
    flags: string[];
  };
  traceId: string;
};

// Supervisor uses envelope to gate aggregation
function aggregateSignals<T>(
  inputs: AgentOutputEnvelope<T>[],
  minConfidence = 0.7
): { highConfidence: AgentOutputEnvelope<T>[]; flagged: AgentOutputEnvelope<T>[] } {
  return inputs.reduce(
    (acc, input) => {
      if (input.confidence.score >= minConfidence && !input.freshness.isStale) {
        acc.highConfidence.push(input);
      } else {
        acc.flagged.push(input);
      }
      return acc;
    },
    { highConfidence: [] as AgentOutputEnvelope<T>[], flagged: [] as AgentOutputEnvelope<T>[] }
  );
}

Genie One's design choice to ground mobile AI capabilities in Unity Catalog and SSO/MFA rather than building mobile-specific permission logic is the right call architecturally, and it's instructive for agentic platform design more broadly. The principle is: governance enforcement belongs at the data and identity layer, not in the agent's prompt or the application's session management. When enforcement lives at the data layer, your observability tools automatically capture access events regardless of which surface triggered the agent invocation — the Unity Catalog audit log doesn't care whether the query came from a web dashboard or a phone. Source: [Take insights anywhere with Genie One on mobile] — Databricks (https://www.databricks.com/blog/take-insights-anywhere-genie-one-mobile)

AIThis connects to a broader pattern in decomposed observability: the more surfaces your agentic system spans, the more critical it becomes that your telemetry anchor is the workflow and agent identity, not the UI session. A trace ID that originates at workflow dispatch and propagates through every tool call — regardless of whether that call happens from a mobile context, a background processor, or a web session — gives you a single reconstruction surface for incident analysis. Without that, you end up with fragmented logs that make it nearly impossible to answer 'what exactly did the agent do on behalf of this user at 2:17 AM when the mobile app retried a failed request?'

// TypeScript: Session-scoped trace propagation across surfaces
import { AsyncLocalStorage } from 'async_hooks';

type AgentContext = {
  workflowId: string;
  traceId: string;
  surface: 'web' | 'mobile' | 'api';
  userId: string;
};

const agentContext = new AsyncLocalStorage<AgentContext>();

// Entry point — set once at invocation boundary
function runWithAgentContext<T>(
  ctx: AgentContext,
  fn: () => Promise<T>
): Promise<T> {
  return agentContext.run(ctx, fn);
}

// Anywhere in the call stack — no prop drilling
function getAgentContext(): AgentContext {
  const ctx = agentContext.getStore();
  if (!ctx) throw new Error('No agent context — is this call inside runWithAgentContext?');
  return ctx;
}

// Tool call automatically picks up trace context
async function queryDataLayer(sql: string): Promise<unknown> {
  const { traceId, workflowId, userId, surface } = getAgentContext();
  // Attach to outgoing request headers / audit log
  return fetch('/data-api/query', {
    method: 'POST',
    headers: {
      'x-trace-id': traceId,
      'x-workflow-id': workflowId,
      'x-user-id': userId,
      'x-surface': surface,
    },
    body: JSON.stringify({ sql }),
  });
}

The '99% is infrastructure' framing from the Data + AI Summit founders panel points at something that's easy to miss when you're deep in framework selection: the agent loop itself is relatively tractable. LangGraph, CrewAI, Strands — they all give you a working reasoning loop. What they don't give you out of the box is the configuration governance (who can change which tool a production agent is allowed to call?), the deployment security (how do you rotate a credential used by a running agent without interrupting in-flight workflows?), and the ongoing evaluation (how do you know your retrieval quality hasn't degraded since last week's index update?). These are platform engineering problems dressed in AI clothing, and they're exactly where senior platform engineers have accumulated leverage that pure ML practitioners haven't. Source: [Stop giving your agents database credentials] — CrewAI (https://blog.crewai.com/stop-giving-your-agents-database-credentials/)

AIThis maps cleanly onto the broader pattern signal from this week: the field is moving from experimentation to hardening. The engineers who built distributed systems, designed service meshes, and wrestled with credential rotation in microservice architectures have a genuine head start on the infrastructure 99% — the challenge is recognizing that the patterns translate, even when the terminology is different. Async guardrail evaluation is just a specialized message filter. Per-phase provenance logging is just structured audit logging with agent context. Human-in-the-loop gates are circuit breakers with a human in the resolution path. The concepts are familiar; the application surface is new.

// TypeScript: Async guardrail filter as a pipeline stage
// Plugs into any agent output stream — not model-specific

type GuardrailResult = {
  passed: boolean;
  policyId: string;
  score: number;
  reason?: string;
};

type GuardrailFilter = (output: string, context: AgentContext) => Promise<GuardrailResult>;

async function applyGuardrails(
  output: string,
  ctx: AgentContext,
  filters: GuardrailFilter[]
): Promise<{ output: string; violations: GuardrailResult[] }> {
  const results = await Promise.all(filters.map(f => f(output, ctx)));
  const violations = results.filter(r => !r.passed);

  if (violations.length > 0) {
    // Emit structured violation events — don't just throw
    violations.forEach(v =>
      emitObservabilityEvent({
        type: 'guardrail.violation',
        policyId: v.policyId,
        score: v.score,
        reason: v.reason,
        traceId: ctx.traceId,
        workflowId: ctx.workflowId,
      })
    );
  }

  return { output, violations };
}

function emitObservabilityEvent(event: Record<string, unknown>): void {
  // Route to your observability bus — OTel, CloudWatch, Datadog, etc.
  console.log(JSON.stringify({ ...event, timestamp: new Date().toISOString() }));
}

The practical case for a local agent stack isn't primarily about running open-weight models in production — it's about having an environment where you can iterate on instrumentation without incurring API costs or hitting rate limits every time you test a new telemetry hook. When you're designing decomposed observability for an LLM pipeline, you need to run the same workflow dozens of times with different failure scenarios injected at different stages: what happens to your trace when the tool call times out? Does your supervisor correctly attribute the failure? Does your anomaly detector fire? None of that is cheap to test against hosted model APIs. Source: [Using Local Coding Agents] — Sebastian Raschka (https://magazine.sebastianraschka.com/p/using-local-coding-agents)

AIThere's also a less-discussed benefit: local stacks let you inspect the full prompt that actually reaches the model at each step of the pipeline, which is critical for validating that your context management and phase-emission logic is working correctly. In a hosted API context, you're often inferring what the model received from its output behavior. Locally, you can log the exact input, the full tool schema provided, and the raw completion — and build your observability assertions against that ground truth. That's the foundation of the draft-then-verify pattern applied to the observability layer itself: verify that your instrumentation is capturing what you think it's capturing, before you trust it in production.

// TypeScript: Local OTel setup for agent harness — same config travels to prod
import { NodeSDK } from '@opentelemetry/sdk-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { Resource } from '@opentelemetry/resources';
import { SEMRESATTRS_SERVICE_NAME } from '@opentelemetry/semantic-conventions';

const isLocal = process.env.NODE_ENV !== 'production';

const sdk = new NodeSDK({
  resource: new Resource({
    [SEMRESATTRS_SERVICE_NAME]: 'agent-pipeline',
    'deployment.environment': isLocal ? 'local' : 'production',
  }),
  traceExporter: new OTLPTraceExporter({
    // Local: Jaeger or Grafana Tempo; prod: same interface, different endpoint
    url: isLocal
      ? 'http://localhost:4318/v1/traces'
      : process.env.OTEL_EXPORTER_OTLP_ENDPOINT!,
  }),
});

sdk.start();

// Simulate tool failure for observability testing
async function simulateToolFailure(failRate: number): Promise<void> {
  if (Math.random() < failRate) {
    throw new Error('simulated_tool_timeout');
  }
}

The practical observability implication of RLHF versus DPO isn't about which training approach is better — it's about knowing what shape of failure to expect in production. An RLHF-trained model that has learned to optimize a proxy reward can produce outputs that look correct by every surface metric (length, format, fluency) while being wrong in a way that only becomes visible when you evaluate against ground truth on your specific task. If your async evaluation filter is scoring for fluency or format compliance, it will miss this class of failure entirely. DPO models tend to have sharper preference boundaries, which means outputs near the edge of their training distribution can shift abruptly in tone or quality — something a rolling variance metric on your evaluation scores can catch if you're watching for it. Source: [How LLMs Learn to Be Helpful (RLHF vs DPO)] — ByteByteGo (https://blog.bytebytego.com/p/how-llms-learn-to-be-helpful-rlhf)

AIThis connects to the guardrail-as-async-filter trend in a specific way: the filters you deploy need to be calibrated to the model's failure mode, not just to general content policy. A platform team that runs multiple models — different providers, different alignment approaches — needs per-model evaluation profiles, not a single shared rubric. Building that into your observability framework now, while your model roster is small, is significantly cheaper than retrofitting it after you've standardized on a scoring approach that misses a whole class of failures for half your deployed models.

// TypeScript: Per-model evaluation profile registry
type AlignmentType = 'rlhf' | 'dpo' | 'unknown';

type EvaluationProfile = {
  alignmentType: AlignmentType;
  primaryRisk: string;
  scoringStrategy: 'semantic' | 'variance' | 'hybrid';
  minPassScore: number;
  varianceWindowSize: number; // rolling window for DPO models
};

const modelProfiles: Record<string, EvaluationProfile> = {
  'anthropic.claude-3-5-sonnet': {
    alignmentType: 'rlhf',
    primaryRisk: 'reward_hacking',
    scoringStrategy: 'semantic',
    minPassScore: 0.82,
    varianceWindowSize: 0,
  },
  'meta.llama3-70b': {
    alignmentType: 'dpo',
    primaryRisk: 'distribution_edge',
    scoringStrategy: 'variance',
    minPassScore: 0.78,
    varianceWindowSize: 20,
  },
};

function getEvaluationProfile(modelId: string): EvaluationProfile {
  return modelProfiles[modelId] ?? {
    alignmentType: 'unknown',
    primaryRisk: 'unknown',
    scoringStrategy: 'hybrid',
    minPassScore: 0.80,
    varianceWindowSize: 10,
  };
}