Holding the Line: How Senior Engineers Keep Architectural Judgment While AI Tooling Accelerates Everything Around Them

The pattern described here is architecturally close to what you'd build for any distributed system where a change is hard to reason about in isolation: instrument first, then verify against live signal. The difference is that the signal consumer is now the agent itself — or more precisely, the human reviewing the agent's work has the signal alongside the diff, not buried in a separate dashboard tab. When a coding agent like Claude Code produces a multi-hundred-line diff from a planning session that included GitHub issues and Slack threads as context, the risk isn't that the code is syntactically wrong; it's that the agent's model of the system doesn't match the system's actual behaviour under load, edge conditions, or concurrent state. Telemetry surfaced at review time — latency distributions, error rates, dependency call graphs — gives you a concrete falsifiability check against the agent's implicit assumptions.

This connects directly to the broader Decomposed Observability trend: you can't govern what you can't see, and for agentic code generation, 'seeing' means instrumenting the agent's output environment, not just the agent itself. Source: [Telemetry-driven development: How to gain confidence in your coding agents' behavior with gcx and Grafana MCP] — Grafana Labs (https://grafana.com/blog/telemetry-driven-development-how-to-gain-confidence-in-your-coding-agents-behavior-with-gcx-and-grafana-mcp/)

// TypeScript: Pull relevant telemetry into PR review context via Grafana MCP
async function fetchReviewSignals(
  services: string[],
  windowMinutes: number
): Promise<ReviewSignal[]> {
  const client = new GrafanaMCPClient({ endpoint: process.env.GRAFANA_MCP_URL });

  return Promise.all(
    services.map(async (svc) => ({
      service: svc,
      errorRate: await client.queryPrometheus(
        `rate(http_requests_total{service="${svc}",status=~"5.."}[${windowMinutes}m])`
      ),
      p99Latency: await client.queryPrometheus(
        `histogram_quantile(0.99, rate(http_request_duration_seconds_bucket{service="${svc}"}[${windowMinutes}m]))`
      ),
      anomalies: await client.getAnomalyAnnotations(svc, windowMinutes),
    }))
  );
}

The cybersecurity eval framework described by Eugene Yan solves a specific problem that anyone building agent evals will hit: if you only score on final outcomes, you get agents that are brittle in novel situations because they learned to pattern-match to success states rather than develop generalizable reasoning chains. Rewarding intermediate steps — the equivalent of 'found the vulnerability' before 'exploited the vulnerability' — forces the evaluation to validate the agent's actual decision process. For a DevOps agent doing infrastructure change validation, this translates to: did the agent correctly identify which services are affected, before it proposes the rollout plan? Did it check for schema compatibility before generating migration SQL? Scoring these as distinct dimensions gives you a much more useful signal than pass/fail on the final artifact.

This pattern also directly addresses the technical debt risk from AI acceleration: when you compress the planning and implementation cycle with agent tooling, you lose visibility into whether the agent's implicit assumptions were sound. Graded intermediate evaluation is one of the few mechanisms that restores that visibility without slowing the pipeline back to human-only speed. Source: [Patterns for Building Cybersecurity Evals] — Eugene Yan (https://eugeneyan.com//writing/cybersecurity-evals/)

// TypeScript: Multi-stage eval runner with graded subtask scoring
interface EvalStage {
  name: string;
  check: (agentState: AgentState) => Promise<boolean>;
  weight: number;
}

async function runGradedEval(
  agent: AgentExecutor,
  task: EvalTask,
  stages: EvalStage[]
): Promise<EvalResult> {
  const agentState = await agent.run(task.input, { sandbox: true });
  const scores: StageScore[] = [];

  for (const stage of stages) {
    const passed = await stage.check(agentState);
    scores.push({ stage: stage.name, passed, weight: stage.weight });
    // Short-circuit only if downstream stages have no meaning without this one
    if (!passed && stage.name === 'problem_identification') break;
  }

  const total = scores.reduce(
    (acc, s) => acc + (s.passed ? s.weight : 0), 0
  );
  return { taskId: task.id, scores, totalScore: total, agentState };
}

The Databricks AI Functions approach is architecturally significant beyond the convenience angle. When you invoke `ai_parse_document` or `ai_extract` inside a SQL query, those calls run under the same identity, permissions, and audit context as the query itself. That means your data steward's existing Unity Catalog policies cover the LLM call — you don't need to build a parallel governance layer for the AI path. For platform engineers, this is the difference between a system that has governance and a system that has governance except for the AI parts, which is the current state of most organisations that bolted LLM pipelines onto existing data warehouses.

AIThe connection to the broader AI acceleration debate is direct: the fastest path to adding AI capabilities is usually also the path that accumulates the most governance debt. In-warehouse functions trade some flexibility (you can't swap to an arbitrary LLM provider mid-query) for a clean governance boundary, which is a reasonable trade for production data platforms where audit requirements are non-negotiable. Source: [Using AI_Functions in Your Data Warehouse: Top Use Cases] — Databricks (https://www.databricks.com/blog/using-aifunctions-your-data-warehouse-top-use-cases)

-- SQL: Using Databricks AI Functions for governed entity extraction
-- Governance model identical to any other warehouse query
SELECT
  ticket_id,
  created_at,
  ai_extract(
    ticket_body,
    ARRAY('service_name', 'error_code', 'severity'),
    'Extract the affected service, error code, and severity from this support ticket'
  ) AS extracted_fields,
  ai_classify(
    ticket_body,
    ARRAY('infrastructure', 'application', 'data', 'security'),
    'Classify this support ticket into one operational domain'
  ) AS domain_classification
FROM support_tickets
WHERE created_at > CURRENT_DATE - INTERVAL 7 DAYS
  AND processed = FALSE;

The experiment described by Giles Edwards-Alexander on martinfowler.com surfaces something that most platform teams haven't fully internalized yet: the economics of AI tooling are directly coupled to the shape of your codebase. Token cost isn't just a billing line item — it's a feedback signal on code complexity. A function that requires three full files of context to understand and modify isn't just hard for humans to reason about; it's expensive for every AI-assisted operation that touches it, across every engineer using that tooling, every day. At scale, that adds up to a cost centre that refactoring can demonstrably reduce.

This creates a novel argument for technical debt remediation that bypasses the usual 'developer velocity' framing that stakeholders often discount. Instead of 'this code is hard to read', you now have 'this code costs $X per AI-assisted modification, and decomposing it brings that to $Y.' That's a conversation that lands differently in an engineering planning session. The implication for AI-accelerated teams is that investing in code health before heavily adopting agent-assisted workflows pays compound returns — not just in maintainability but in the unit economics of every subsequent AI interaction. Source: [The Economic Benefit of Refactoring] — Giles Edwards-Alexander / martinfowler.com (https://martinfowler.com/articles/exploring-gen-ai/refactoring-economic-benefit.html)

// TypeScript: Token cost instrumentation around AI-assisted code operations
// Tracks cost per function scope to identify refactoring candidates

async function measureAgentOperationCost(
  targetFunction: FunctionMetadata,
  operation: 'explain' | 'modify' | 'test_generate'
): Promise<CostMeasurement> {
  const contextTokens = await estimateContextTokens(targetFunction);
  const { result, usage } = await agentClient.invoke({
    operation,
    target: targetFunction,
    trackUsage: true,
  });

  return {
    functionId: targetFunction.id,
    linesOfCode: targetFunction.loc,
    contextTokens,
    promptTokens: usage.prompt_tokens,
    completionTokens: usage.completion_tokens,
    totalCost: calculateCost(usage),
    costPerLoc: calculateCost(usage) / targetFunction.loc,
    operation,
    timestamp: new Date().toISOString(),
  };
}

The Amazon Nova Forge BYOO capability hands you a double-edged tool: you get to define exactly what the model optimizes for across multi-turn conversations, but that means every assumption baked into your reward function becomes a real-world property of the model's behaviour. This is the RL version of the same problem you see in agentic pipeline design — when you decompose a complex task into steps and score each step, you have to be precise about what 'correct' means at each step, or the agent learns to satisfy your scoring rubric rather than the underlying goal. The training curve looks fine. The model ships. The problem shows up in production.

For engineers building agentic systems, the lesson is that reward function design deserves the same review discipline as a production API contract or a database schema migration. It's not a configuration detail; it's an architectural decision with long-term consequences. [AI Synthesis] The connection to governance checkpoints is direct: just as you'd gate a schema migration on review by someone who understands the downstream consumers, you should gate reward function changes on review by someone who can reason about emergent model behaviour across the full task distribution — not just the eval set. Source: [Custom reward functions for multi-turn reinforcement learning with Amazon Nova Forge] — AWS Machine Learning Blog (https://aws.amazon.com/blogs/machine-learning/custom-reward-functions-for-multi-turn-reinforcement-learning-with-amazon-nova-forge/)

# Python: Multi-turn reward function with decomposed step scoring
# Each turn scored independently; final reward is weighted aggregate

def score_multi_turn_conversation(
    turns: list[dict],
    task_spec: TaskSpec
) -> RewardSignal:
    step_scores = []

    for i, turn in enumerate(turns):
        step_score = StepScore(
            turn_index=i,
            # Did the model correctly interpret the user's intent?
            intent_alignment=score_intent_alignment(turn, task_spec),
            # Did the model stay within the defined tool use boundaries?
            tool_constraint_adherence=score_tool_adherence(turn, task_spec.allowed_tools),
            # Did the model's reasoning chain match the expected problem structure?
            reasoning_validity=score_reasoning_chain(turn),
        )
        step_scores.append(step_score)

    # Weight later turns more heavily — early errors that compound are penalized
    weights = [0.5 + (i / len(turns)) * 0.5 for i in range(len(turns))]
    total = sum(s.composite() * w for s, w in zip(step_scores, weights))

    return RewardSignal(
        total_reward=total / sum(weights),
        step_breakdown=step_scores,
        audit_log={
            "task_id": task_spec.id,
            "turn_count": len(turns),
            "reward_version": task_spec.reward_version
        }
    )

Lilian Weng's careful treatment of scaling laws is a useful corrective to the instinct — common in fast-moving teams — to escalate model size as the default response to agent capability gaps. The power-law relationship between compute and loss is well-established and empirically robust: you can predict, with reasonable accuracy, how much a model will improve for a given increase in size or training data. That predictability is valuable not because it tells you to scale up, but because it gives you a quantitative basis for deciding when not to. If your agentic pipeline is underperforming, and the model is already large relative to your task's data distribution, more scale is probably not the answer — better task decomposition, more targeted evaluation data, or architectural changes to how context is passed between steps are more likely to move the needle.

AIFor platform engineers making infrastructure decisions, this connects to the cloud-agnostic principle: the right model for an agentic workload is the smallest model that reliably handles the task at the required quality bar, because that's the one with the best cost, latency, and operational profile. Scaling laws give you the empirical framework to make that argument with numbers rather than intuition. Source: [Scaling Laws, Carefully] — Lilian Weng (https://lilianweng.github.io/posts/2026-06-24-scaling-laws/)

// TypeScript: Model selection guard based on task complexity classification
// Prevents reflexive escalation to larger models when task design is the bottleneck

const MODEL_TIERS = {
  lightweight: 'nova-micro',
  standard: 'nova-lite',
  reasoning: 'nova-pro',
} as const;

function selectModelTier(
  task: AgentTask,
  perfHistory: TaskPerfHistory
): keyof typeof MODEL_TIERS {
  const recentFailureRate = perfHistory.failureRateLast100(task.type);
  const avgContextTokens = perfHistory.avgContextTokens(task.type);

  // High failure rate + small context = task design problem, not model size problem
  if (recentFailureRate > 0.15 && avgContextTokens < 2000) {
    console.warn(`Task ${task.type}: high failure rate with small context — investigate decomposition before escalating model tier`);
    return 'standard';
  }

  if (task.requiresMultiStepReasoning && avgContextTokens > 8000) return 'reasoning';
  if (task.requiresMultiStepReasoning) return 'standard';
  return 'lightweight';
}