Outcome.
Did the end state match the objective? Necessary, insufficient, and the only thing most agent benchmarks report.
The answer was right.
The path was not.
An agent that reaches a correct result by reading records it should not have touched has failed. Agent evaluation scores the whole trajectory.
READ THE DEFINITION ↓TRAJECTORY · STEPS · RECOVERY
01 / DEFINITION
The short answer first.What is AI agent evaluation?
AI agent evaluation measures a multi-step system by its whole trajectory: the actions taken, their order, the data touched and the final state produced. Unlike single-response evaluation, it has to account for side effects, because an agent changes the world as it works.
Three properties make agent evaluation harder than model evaluation. The action space is combinatorial, so exhaustive coverage is impossible. Side effects are real, so evaluation needs a sandbox with realistic state rather than a static test set. And success is multidimensional: the same end state can be reached efficiently, wastefully, or through a path that breached a boundary.
02 / WHAT TO SCORE
Five dimensions, measured separately.Path validity.
Was every action permitted and appropriate? An agent that read records outside its scope failed, whatever the outcome.
Efficiency.
Steps, tokens and tool calls against a reasonable baseline. Waste correlates with confusion and predicts failure on harder instances.
Recovery.
What happened after a tool error or an empty result? Retrying identically until a step limit is the most common failure pattern in practice.
Refusal.
Did the agent stop when it could not proceed safely, or did it improvise? Correct refusal is a capability and should be scored as one.
Path validity is where evaluation meets security. An agent that produces correct answers while routinely reading beyond its scope will eventually be given a task where that behaviour is harmful. Scoring the path makes the tendency visible while it is still cheap, which is why agent evaluation belongs alongside AI agent permissions rather than in a separate quality workstream.
03 / BUILDING THE HARNESS
What an agent test environment requires.| Component | Purpose | Common shortcut and its cost |
|---|---|---|
| Sandboxed state | Realistic data the agent can safely modify | Testing against production; unrepeatable and risky |
| Deterministic tools | Same call, same response, every run | Live APIs; failures cannot be attributed to the agent |
| Injected faults | Timeouts, empty results, malformed responses | Happy-path only; recovery behaviour never measured |
| Adversarial content | Documents carrying hidden instructions | Clean corpora; injection resistance untested |
| Trajectory log | Every step, input and decision, replayable | Final-answer logging; no way to diagnose a failure |
Fault injection is the component with the best return and the lowest adoption. Most agent failures observed in production are not reasoning failures but handling failures: a tool returned an error, the agent treated the error text as data, and the trajectory degraded from there. A harness that never returns an error has not tested the majority of real failure modes.
04 / FROM EVALUATION TO AUTONOMY
What a score should be allowed to buy.Autonomy is granted,
not achieved.
Agent evaluation produces a distribution of behaviour, not a guarantee. The useful output is not a pass or fail but a mapping from measured evidence to permitted authority, per capability: this action type has a strong record and may proceed with logging; that one is proposal-only until a sustained record exists.
Privilege AI applies the same structure to model supervision, where authority is set per check rather than per model — 25 of MKC2's 45 checks carry report-only authority and one is refused outright. The agent case is the same argument applied to actions instead of findings, and it is developed in AI agent governance and AI model supervision.
A supervision model can also review agent actions directly, which is part of what MKC2 is built to do: evaluate AI-generated work and the actions of AI agents, in review-only mode.
05 / QUESTIONS
Asked before an agent is given more room.How is agent evaluation different from model evaluation?
+
The unit of analysis. Model evaluation scores a response; agent evaluation scores a sequence of actions with side effects. An agent can produce a correct answer through an impermissible path, and only trajectory-level scoring will show it.
How many test scenarios does an agent need?
+
Coverage should be organised by failure mode rather than by count. A few dozen scenarios spanning normal operation, tool failure, ambiguous instructions, adversarial content and out-of-scope requests is more informative than hundreds of happy-path variations.
Can agents be evaluated in production?
+
Partially, through logged trajectories and sampled human review of proposed actions. What production cannot provide is a controlled comparison, because state differs on every run. Both are needed: the sandbox for attribution, production for realism. See AI agent monitoring.
Should an agent be scored on cost?
+
Yes. Step count, token use and wall-clock time are quality signals as well as budget signals: a run that takes four times the baseline steps is usually confused, and the excess predicts failure on harder instances.
Who defines what a correct trajectory is?
+
The domain expert who owns the process, working with whoever owns access control. The first defines what should happen; the second defines what is permitted. Writing both down is most of the evaluation design.
Score the path,
not just the answer.
Privilege AI evaluates AI-generated work and the actions of AI agents, with authority set by measured evidence.