Cut the Fat, Keep the SLO: Latency and Cost Optimization Patterns for Large-Scale Agentic Recommendation Pipelines

The sports intelligence architecture described by Databricks is a clean illustration of a principle that applies directly to agentic recommendation systems: the agent itself should be a thin consumer of pre-processed signals, not a data wrangler. When Hawk-Eye SkeleTRACK is generating 20,000 data points per second, the cost of letting an agent perform ad-hoc retrieval over raw tracking data at inference time is prohibitive — both in latency and in token spend if you're materializing that context into a prompt. The pattern that scales is building a governed feature layer that continuously reduces high-velocity raw data into keyed, agent-consumable signals, so the agent's tool call hits a fast lookup rather than a streaming join. Source: [The Rise of Sports Intelligence: How the Lakehouse Turns Tracking Data into Competitive Advantage] — (https://www.databricks.com/blog/rise-sports-intelligence-how-lakehouse-turns-tracking-data-competitive-advantage)

AIThis maps directly onto the cost problem in recommendation agents: if your agent is calling an embedding search or a vector retrieval tool on every request against the full corpus, you're paying both the retrieval latency and the downstream LLM cost to process a wide context window. Materializing rolling aggregates — player load scores, recent form vectors, injury risk indices — into a keyed store that the agent can hit with a single fast lookup is the same pattern as sports analytics pre-computation, just applied to your recommendation context. The governance angle matters too: when a recommendation is challenged, you want to be able to point at the feature values that drove it, not reconstruct them from raw event logs.

// Agent tool: fast keyed lookup instead of ad-hoc retrieval
const getEntityContext = tool(
  async ({ entityId }: { entityId: string }) => {
    // Sub-millisecond keyed read from materialized feature store
    // No vector search, no streaming join, no raw data wrangling
    const features = await featureStore.get(entityId);
    return {
      recentFormScore: features.rolling_7d_performance,
      loadRiskIndex: features.cumulative_load_pct,
      injuryRiskTier: features.biomechanical_risk_tier,
      asOf: features.materialized_at_utc,
    };
  },
  {
    name: 'get_entity_context',
    description: 'Fetch pre-aggregated recommendation context for an entity. Always prefer this over raw data retrieval.',
    schema: z.object({ entityId: z.string() }),
  }
);

The EKS Auto Mode improvements — faster node readiness, smarter scaling triggers, and DNS optimization for new pods — address a class of latency that lives below the agent orchestration layer but shows up in your end-to-end SLO numbers. When you're running a recommendation agent fleet that needs to scale horizontally under load, the time between 'Kubernetes decides to add a node' and 'that node is actually serving traffic' is a floor on your burst latency. If that floor is 90 seconds, no amount of prompt optimization or model selection helps your p99 during a traffic spike. Source: [Faster nodes, smarter scaling: What's new inside Amazon Elastic Kubernetes Service (Amazon EKS) Auto Mode] — (https://aws.amazon.com/blogs/containers/faster-nodes-smarter-scaling-whats-new-inside-amazon-elastic-kubernetes-service-amazon-eks-auto-mode/)

AIThe actionable posture here for a platform team is to instrument the full provisioning path — not just agent execution time — in your distributed traces. A span that covers 'request received' to 'first agent tool call initiated' will surface infrastructure overhead that agent-layer observability alone misses. Combined with pre-warming strategies for your most latency-sensitive agent workloads, treating infrastructure provisioning as part of the SLO conversation rather than a separate ops concern is how you actually hit your targets under real load patterns.

// Instrument the full provisioning path, not just agent execution
import { trace, SpanStatusCode } from '@opentelemetry/api';

const tracer = trace.getTracer('recommendation-agent');

async function handleRecommendationRequest(req: RecommendationRequest) {
  return tracer.startActiveSpan('recommendation.full_path', async (rootSpan) => {
    // This timestamp captures infrastructure overhead BEFORE agent starts
    rootSpan.setAttribute('request.received_at', Date.now());

    const infraSpan = tracer.startSpan('infra.pod_readiness_check');
    await waitForPodWarm(); // detect if this pod is freshly scheduled
    infraSpan.setAttribute('pod.was_cold', podIsCold());
    infraSpan.end();

    // Agent execution starts here — now you can diff infra vs agent latency
    return tracer.startActiveSpan('agent.execute', async (agentSpan) => {
      const result = await runRecommendationAgent(req);
      agentSpan.setAttribute('agent.tool_calls', result.toolCallCount);
      agentSpan.end();
      rootSpan.end();
      return result;
    });
  });
}

The Kubernetes Device Management Working Group's Dynamic Resource Allocation feature is solving a problem that hits hard when you operate multiple inference services — a recommendation agent, an embedding pipeline, a reranker — on shared GPU infrastructure. The traditional model of declaring GPU resource requests at pod scheduling time creates a binary choice: overprovision to handle peak load, or risk resource contention. DRA introduces structured hardware claims that can be satisfied, modified, and released with finer granularity, including after pod startup — which maps well to the reality that an agent doing retrieval-augmented recommendation has very different GPU pressure during context assembly versus during generation. Source: [Spotlight on WG Device Management] — (https://kubernetes.io/blog/2026/06/24/wg-device-management-spotlight-2026/)

AIFor a platform team running agentic recommendation systems, the near-term implication is architectural: if you're currently deploying your embedding model, your reranker, and your generation model as separate pods each holding dedicated GPU slices, DRA gives you a path to pooling that capacity and scheduling against actual demand rather than worst-case estimates. The latency benefit is secondary to the cost benefit here — but reduced overprovisioning also means faster cluster bin-packing, which improves your average node utilization and, indirectly, your burst scaling headroom.

# Kubernetes DRA ResourceClaim for shared inference GPU allocation
# Instead of static nvidia.com/gpu: 1 in every pod spec
apiVersion: resource.k8s.io/v1alpha3
kind: ResourceClaim
metadata:
  name: recommendation-agent-gpu-claim
spec:
  devices:
    requests:
    - name: inference-gpu
      deviceClassName: gpu.nvidia.com
      count: 1
      # Release back to pool when agent is in idle/waiting state
      # rather than holding the slice for the pod's full lifetime
  allocationMode: WaitForFirstConsumer

The Huntington Bank redaction project — systematically processing hundreds of millions of documents to locate and remove sensitive customer data — is structurally identical to a class of agentic tool execution problems: long-horizon, high-volume, async, with zero tolerance for undetected failures. The engineering constraints are the same whether your agent is redacting PII or generating personalized recommendations: you need per-item outcome tracking, resumable execution state, and a governance layer that can answer 'what happened to record X' without replaying the entire job. Source: [Huntington Bank: Redacting sensitive data from 400M+ documents with AWS] — (https://aws.amazon.com/blogs/machine-learning/huntington-bank-redacting-sensitive-data-from-400m-documents-with-aws/)

AIFor platform teams running agentic recommendation pipelines, the governance lesson here is about where you draw the audit boundary. If your agent produces a recommendation and your logging captures the final output but not the intermediate tool calls, retrieved context, and scoring signals that produced it, you have a system that's hard to debug and impossible to audit. The Huntington pattern — treat every processed item as an auditable unit with its own outcome record — maps directly onto structuring each recommendation agent run as a persisted trace with tool call inputs, outputs, and decision context attached. This is more expensive in storage than aggregate logging, but it's the only architecture that actually supports governance at scale.

// Per-record outcome logging for auditable async agent tool execution
interface AgentToolOutcome {
  recordId: string;
  jobId: string;
  toolName: string;
  toolInputSnapshot: Record<string, unknown>;
  toolOutputSnapshot: Record<string, unknown>;
  agentDecision: string;
  executedAt: string; // ISO 8601
  durationMs: number;
  checkpointKey: string; // for resumability
}

async function executeWithAudit(
  recordId: string,
  toolCall: () => Promise<ToolResult>
): Promise<ToolResult> {
  const start = Date.now();
  const result = await toolCall();

  // Write audit record BEFORE acknowledging success to queue
  await auditStore.append({
    recordId,
    jobId: currentJobId,
    toolName: result.toolName,
    toolInputSnapshot: result.input,
    toolOutputSnapshot: result.output,
    agentDecision: result.decision,
    executedAt: new Date().toISOString(),
    durationMs: Date.now() - start,
    checkpointKey: `${currentJobId}:${recordId}`,
  } satisfies AgentToolOutcome);

  return result;
}