The production question asks whether the system took the right action, for the right reason, within the right boundary.
A tool-using agent can change an order, query private records, send a message, update a workflow, or trigger another system. Once that happens, fluency becomes one small part of the operating contract.
Begin with the action boundary
I start by listing what the agent may do, what it must never do, and which actions require a human confirmation. This sounds simple, but it exposes ambiguity early: teams often agree on the happy path and disagree on authority, exceptions, and recovery.
- Which tools can the agent invoke?
- Which data can each tool read or write?
- What evidence must support an action?
- What requires confirmation, escalation, or refusal?
- How is an incorrect action detected and reversed?
Separate the failure classes
A single “task success” score is rarely enough. A failed action may begin with the wrong retrieved context, a planning error, malformed arguments, an authorization problem, a tool timeout, or an unsafe recovery attempt. Those failures have different owners and different fixes.
Evaluation becomes useful when a failing test tells an engineer where to look next.
A practical taxonomy usually includes task interpretation, retrieval, reasoning or planning, tool selection, argument construction, permission enforcement, result interpretation, response support, and recovery behavior.
Build a release gate around representative tasks
I prefer a small set of carefully chosen tasks over a large pile of generic prompts. The set should include ordinary work, high-value work, known edge cases, ambiguous requests, unavailable tools, insufficient evidence, and attempts to cross a permission boundary.
For each task, the test should capture more than the final response. It should retain the evidence selected, tool calls attempted, arguments used, permission decisions, tool results, state changes, and any escalation.
Measure what changes the release decision
The most useful metrics depend on the workflow, but the release decision commonly needs:
- task success for representative and critical slices;
- correct tool selection and valid arguments;
- unsupported-action and permission-violation rates;
- answer or action support from available evidence;
- appropriate refusal and human escalation;
- recovery success after predictable tool and data failures;
- latency and cost across the full workflow, including each model call.
Keep the gate connected to production
A release suite becomes stale when it is separated from real incidents. Production traces, user corrections, failed escalations, and near misses should feed new evaluation cases. The loop is simple: observe, classify, reproduce, add the case, fix the system, and keep the regression.
The goal is controlled autonomy: capabilities and limits explicit enough for a team to decide where action is justified.