Instrumenting Function Calls in Production Agents: From Blind Invocations to Observable, Governed Tool Use

The Grafana approach to Cypress test monitoring makes a point that lands directly in agentic observability: terminal output and per-run CI logs are fundamentally the wrong unit of analysis when you're trying to find patterns. The same is true of function call logs emitted per agent session. What you need is a persistent time-series store — whether that's Prometheus, InfluxDB, or a custom PostgreSQL schema with a timestamped invocations table — that accumulates tool call events across runs, tagged by tool name, model ID, agent node, session ID, and outcome. Then you can actually ask the questions that matter: is this tool's error rate trending upward over the last 48 hours? Did latency spike after a schema update? Which agent nodes are generating outlier invocation counts? Source: [How to monitor Cypress tests with Grafana Cloud] — (https://grafana.com/blog/how-to-monitor-cypress-tests-with-grafana-cloud/)

AIThe pattern from last week's Per-Cycle Integrity Hashing work connects here too — if you're hashing state at cycle boundaries, the function call payload and its return value are precisely the artifacts you want to include in that hash. That makes your invocation telemetry dual-purpose: operational observability and provenance integrity in a single instrumentation pass.

// TypeScript middleware for function call telemetry in an agentic tool registry
import { Counter, Histogram } from 'prom-client';

const toolInvocations = new Counter({
  name: 'agent_tool_invocations_total',
  help: 'Total tool invocations by agent node and outcome',
  labelNames: ['tool_name', 'agent_node', 'model_id', 'outcome'],
});

const toolLatency = new Histogram({
  name: 'agent_tool_latency_seconds',
  help: 'Tool call latency distribution',
  labelNames: ['tool_name', 'agent_node'],
  buckets: [0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10],
});

export function instrumentedTool<TArgs, TResult>(
  toolName: string,
  agentNode: string,
  modelId: string,
  fn: (args: TArgs) => Promise<TResult>
): (args: TArgs) => Promise<TResult> {
  return async (args: TArgs): Promise<TResult> => {
    const end = toolLatency.startTimer({ tool_name: toolName, agent_node: agentNode });
    try {
      const result = await fn(args);
      toolInvocations.inc({ tool_name: toolName, agent_node: agentNode, model_id: modelId, outcome: 'success' });
      return result;
    } catch (err) {
      toolInvocations.inc({ tool_name: toolName, agent_node: agentNode, model_id: modelId, outcome: 'error' });
      throw err;
    } finally {
      end();
    }
  };
}

HashiCorp's move toward pre-written, HCL-native policy templates in Terraform is a signal worth taking seriously for tool governance in agentic systems. The friction problem is real: teams know they should enforce policies on what their agents can call and how, but writing Rego or custom validation logic from scratch is expensive enough that it gets deferred indefinitely. The answer is the same pattern Terraform is applying — ship opinionated, pre-built policies for the common cases, and let teams override or extend rather than build from nothing. For agentic tool registries, that means policies like 'all tools accessing customer data must declare a data_classification label' or 'state-mutating tools require a human-approval annotation when called outside business hours' — enforced declaratively at registration, not buried in agent code. Source: [Simplify compliance with the native pre-written policy experience in HCP Terraform] — (https://www.hashicorp.com/blog/simplify-compliance-with-a-native-pre-written-policy-experience-in-terraform)

AIThis connects directly to the provenance integrity trend from last week: if you're capturing per-cycle state hashes, the tool's schema version and policy evaluation result at call time should be part of that hash input. That way your integrity records reflect not just what state the agent was in, but what governance context governed each tool invocation — which is exactly what an auditor would want to reconstruct after the fact.

// Declarative tool registration with policy annotations
interface ToolPolicy {
  dataClassification: 'public' | 'internal' | 'customer-pii';
  mutatesState: boolean;
  requiresHumanApproval: boolean;
  schemaVersion: string; // semver
  allowedAgentNodes: string[];
}

interface ToolRegistration<TArgs, TResult> {
  name: string;
  description: string;
  argsSchema: object; // JSON Schema
  policy: ToolPolicy;
  handler: (args: TArgs) => Promise<TResult>;
}

function registerTool<TArgs, TResult>(
  registry: Map<string, ToolRegistration<TArgs, TResult>>,
  tool: ToolRegistration<TArgs, TResult>
): void {
  // Policy gate: customer-pii tools must declare approval requirement
  if (tool.policy.dataClassification === 'customer-pii' && !tool.policy.requiresHumanApproval) {
    throw new Error(
      `Tool '${tool.name}' accesses customer PII but does not declare requiresHumanApproval: true`
    );
  }
  if (tool.policy.mutatesState && tool.policy.allowedAgentNodes.length === 0) {
    throw new Error(
      `State-mutating tool '${tool.name}' must declare explicit allowedAgentNodes`
    );
  }
  registry.set(tool.name, tool);
  console.log(`Registered tool: ${tool.name}@${tool.policy.schemaVersion}`);
}

Kubernetes v1.37's graduation of Pod-Level Resource Managers to Beta is infrastructure news, but for anyone running high-volume agentic workers it has a direct operational implication. Before this change, when you had an agent pod with multiple containers — the main worker, an OTel collector sidecar, maybe a tool proxy — each container's CPU and memory placement was negotiated independently. That means your worker could end up on NUMA node 0 while its sidecar lands on NUMA node 1, and every memory operation crossing that boundary adds latency. At a few hundred agent sessions per node that's noise; at thousands of concurrent function call invocations it becomes a measurable floor on your p99. Source: [Kubernetes v1.37: Pod-Level Resource Managers graduated to Beta] — (https://kubernetes.io/blog/2026/09/15/kubernetes-v1-37-pod-level-resource-managers-beta/)

AIThe operational pattern worth building toward is treating your agent worker pods the same way you'd treat a latency-sensitive JVM service: declare CPU and memory at the pod boundary using `.spec.resources`, pin to NUMA-local nodes, and instrument the before/after on function call tail latency. The observability investment from block_1 — your tool call latency histograms — gives you exactly the measurement surface you need to validate whether topology placement is actually moving the needle in your environment.

# Kubernetes Pod spec using pod-level resource declarations (v1.37+ Beta)
apiVersion: v1
kind: Pod
metadata:
  name: agent-worker
  annotations:
    topology-manager-policy: single-numa-node
spec:
  # Pod-level resource declaration — enables co-located NUMA placement
  resources:
    requests:
      cpu: "4"
      memory: "8Gi"
    limits:
      cpu: "4"
      memory: "8Gi"
  containers:
    - name: agent-worker
      image: agent-worker:latest
      # Container-level resources omitted — inherited from pod spec
    - name: otel-collector
      image: otel/opentelemetry-collector:latest
      # Shares NUMA node with agent-worker — no cross-node memory hops

Kimi K3's availability on Amazon Bedrock represents the practical realization of a pattern that's been theoretical for most teams: genuinely mixing open-weight and frontier models in the same agentic pipeline, with Bedrock providing a unified invocation surface. The economics argument for doing this is straightforward — if a capable open-weight model can handle the high-frequency, lower-complexity tool calls in your system, you route those there and reserve the frontier model budget for reasoning-heavy operations. But the engineering consequence is that your tool schemas now need to be validated against multiple model families, not just the one you built against. Source: [Introducing Kimi K3 on Amazon Bedrock] — (https://aws.amazon.com/blogs/machine-learning/introducing-kimi-k3-on-amazon-bedrock/)

AIThis is where the governance pattern from block_2 becomes essential infrastructure rather than nice-to-have: if every tool registration carries a schema version and the invocation telemetry records which model ID called it, you have the data you need to run a model compatibility matrix. Plot error rate by (tool_name, schema_version, model_id) and you get a clear picture of where your schemas are model-specific versus genuinely portable. That's the kind of production observability data that makes the difference between a confident multi-model rollout and a game of whack-a-mole with schema edge cases.

// Multi-model tool call router with schema compatibility validation
const MODEL_PROFILES: Record<string, { maxNestedDepth: number; supportsEnumArrays: boolean }> = {
  'anthropic.claude-3-5-sonnet': { maxNestedDepth: 5, supportsEnumArrays: true },
  'moonshot.kimi-k3': { maxNestedDepth: 3, supportsEnumArrays: false },
};

function validateSchemaForModel(
  schema: object,
  modelId: string
): { valid: boolean; violations: string[] } {
  const profile = MODEL_PROFILES[modelId];
  if (!profile) return { valid: false, violations: [`Unknown model profile: ${modelId}`] };

  const violations: string[] = [];
  const depth = measureSchemaDepth(schema);

  if (depth > profile.maxNestedDepth) {
    violations.push(`Schema nesting depth ${depth} exceeds model limit ${profile.maxNestedDepth}`);
  }
  if (!profile.supportsEnumArrays && containsEnumArrays(schema)) {
    violations.push(`Model ${modelId} does not reliably handle enum array arguments`);
  }
  return { valid: violations.length === 0, violations };
}

function measureSchemaDepth(schema: object, depth = 0): number {
  if (typeof schema !== 'object' || schema === null) return depth;
  return Math.max(...Object.values(schema).map(v => measureSchemaDepth(v as object, depth + 1)));
}

function containsEnumArrays(schema: object): boolean {
  return JSON.stringify(schema).includes('"type":"array"') &&
    JSON.stringify(schema).includes('"enum"');
}

The Kyverno proxy injection pattern for EKS Fargate is architecturally interesting because it solves a constraint that every platform engineer recognizes: when the infrastructure surface isn't yours to configure, you need to push control-plane enforcement up into the Kubernetes admission layer. For agentic systems running on Fargate or any managed node substrate, this maps directly to the challenge of ensuring every tool's outbound HTTP call passes through a governed egress path. A Kyverno ClusterPolicy that mutates agent worker pods to inject an egress proxy sidecar — with environment variables pointing tool HTTP clients at the proxy — can be rolled out cluster-wide without any tool implementation changes. Source: [Automate proxy injection for Amazon EKS on AWS Fargate using Kyverno] — (https://aws.amazon.com/blogs/containers/automate-proxy-injection-for-amazon-eks-on-aws-fargate-using-kyverno/)

AIThe thread connecting this to the broader digest is that tool call observability has two planes: the data plane (what the tool returned, how long it took, whether it failed) covered in block_1, and the network governance plane covered here. The proxy sidecar is also your cheapest egress logging layer — you get a full record of every outbound URL called by every tool, correlated with the pod identity, without adding a single line to any tool's implementation. Feed that into your time-series store alongside your Prometheus metrics and you have a genuinely complete picture of what your agents are actually doing in the world.

# Kyverno ClusterPolicy: inject egress proxy env into all agent worker pods
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: inject-agent-egress-proxy
spec:
  rules:
    - name: add-proxy-env
      match:
        any:
          - resources:
              kinds:
                - Pod
              selector:
                matchLabels:
                  role: agent-worker
      mutate:
        patchStrategicMerge:
          spec:
            initContainers:
              - name: proxy-readiness-check
                image: busybox:latest
                command: ['sh', '-c', 'until nc -z proxy-service 3128; do sleep 1; done']
            containers:
              - (name): "*"
                env:
                  - name: HTTP_PROXY
                    value: "http://proxy-service.infra:3128"
                  - name: HTTPS_PROXY
                    value: "http://proxy-service.infra:3128"
                  - name: NO_PROXY
                    value: "localhost,127.0.0.1,cluster.local"