Privilege AIMKC2 results ↗

AI agent evaluation

An agent that reaches a correct result by reading records it should not have touched has failed. Agent evaluation scores the whole trajectory.

READ THE DEFINITION ↓TRAJECTORY · STEPS · RECOVERY

01 / DEFINITION

The short answer first.

What is AI agent evaluation?

AI agent evaluation measures a multi-step system by its whole trajectory: the actions taken, their order, the data touched and the final state produced. Unlike single-response evaluation, it has to account for side effects, because an agent changes the world as it works.

Three properties make agent evaluation harder than model evaluation. The action space is combinatorial, so exhaustive coverage is impossible. Side effects are real, so evaluation needs a sandbox with realistic state rather than a static test set. And success is multidimensional: the same end state can be reached efficiently, wastefully, or through a path that breached a boundary.

WORKED EXAMPLE

Evaluate the action sequence behind a successful answer

A scheduling agent says it moved a service appointment from Thursday to Friday. The text is correct only if the booking system contains the new appointment, the old slot has been released and the customer received the intended confirmation. An evaluation that reads only the final chat message can miss a duplicate booking or a change applied to the wrong customer.

Specify state before scoring the transcript

Create a test account with an existing Thursday reservation. Give the agent permission to change that reservation and no permission to modify another account. Run the task against a disposable scheduling service and record the tool calls, arguments, results and final bookings. Grade the resulting state independently of the agent's explanation. A polished confirmation cannot compensate for an appointment that remains on Thursday.

Now make the booking service accept the update but delay its response. The agent may retry after a timeout. Check that the retry recognises the original operation and does not reserve Friday twice. Repeat with a genuine rejection, where the expected result is an unchanged booking and a useful explanation rather than a fabricated confirmation.

Include a conflicting request that names another customer's booking. The appropriate outcome is a safe refusal or escalation, even if the system could technically reach the record. Keep task completion, correct refusal, side effects and recovery as separate results. These distinctions tell the release owner whether to improve planning, tool design, permissions or the wording presented to the customer.

02 / WHAT TO SCORE

01

Outcome.

Did the end state match the objective? Necessary, insufficient, and the only thing most agent benchmarks report.

02

Path validity.

Was every action permitted and appropriate? An agent that read records outside its scope failed, whatever the outcome.

03

Efficiency.

Steps, tokens and tool calls against a reasonable baseline. Waste correlates with confusion and predicts failure on harder instances.

04

Recovery.

What happened after a tool error or an empty result? Retrying identically until a step limit is the most common failure pattern in practice.

05

Refusal.

Did the agent stop when it could not proceed safely, or did it improvise? Correct refusal is a capability and should be scored as one.

Path validity is where evaluation meets security. An agent that produces correct answers while routinely reading beyond its scope will eventually be given a task where that behaviour is harmful. Scoring the path makes the tendency visible while it is still cheap, which is why agent evaluation belongs alongside AI agent permissions rather than in a separate quality workstream.

03 / BUILDING THE HARNESS

The components of an agent evaluation harness, and the shortcut each one tempts teams into.
ComponentPurposeCommon shortcut and its cost
Sandboxed stateRealistic data the agent can safely modifyTesting against production; unrepeatable and risky
Deterministic toolsSame call, same response, every runLive APIs; failures cannot be attributed to the agent
Injected faultsTimeouts, empty results, malformed responsesHappy-path only; recovery behaviour never measured
Adversarial contentDocuments carrying hidden instructionsClean corpora; injection resistance untested
Trajectory logEvery step, input and decision, replayableFinal-answer logging; no way to diagnose a failure

Fault injection is the component with the best return and the lowest adoption. Most agent failures observed in production are not reasoning failures but handling failures: a tool returned an error, the agent treated the error text as data, and the trajectory degraded from there. A harness that never returns an error has not tested the majority of real failure modes.

04 / FROM EVALUATION TO AUTONOMY

Autonomy is granted,
not achieved.

What a score should be allowed to buy.

Agent evaluation produces a distribution of behaviour, not a guarantee. The useful output is not a pass or fail but a mapping from measured evidence to permitted authority, per capability: this action type has a strong record and may proceed with logging; that one is proposal-only until a sustained record exists.

Privilege AI applies the same structure to model supervision, where authority is set per check rather than per model — 25 of MKC2's 45 checks carry report-only authority and one is refused outright. The agent case is the same argument applied to actions instead of findings, and it is developed in AI agent governance and AI model supervision.

A supervision model can also review agent actions directly, which is part of what MKC2 is built to do: evaluate AI-generated work and the actions of AI agents, in review-only mode.

05 / QUESTIONS

How is agent evaluation different from model evaluation?

The unit of analysis. Model evaluation scores a response; agent evaluation scores a sequence of actions with side effects. An agent can produce a correct answer through an impermissible path, and only trajectory-level scoring will show it.

How many test scenarios does an agent need?

Coverage should be organised by failure mode rather than by count. A few dozen scenarios spanning normal operation, tool failure, ambiguous instructions, adversarial content and out-of-scope requests is more informative than hundreds of happy-path variations.

Can agents be evaluated in production?

Partially, through logged trajectories and sampled human review of proposed actions. What production cannot provide is a controlled comparison, because state differs on every run. Both are needed: the sandbox for attribution, production for realism. See AI agent monitoring.

Should an agent be scored on cost?

Yes. Step count, token use and wall-clock time are quality signals as well as budget signals: a run that takes four times the baseline steps is usually confused, and the excess predicts failure on harder instances.

Who defines what a correct trajectory is?

The domain expert who owns the process, working with whoever owns access control. The first defines what should happen; the second defines what is permitted. Writing both down is most of the evaluation design.

WORK WITH PRIVILEGE AI

How we can help

Talk with Privilege AI

Describe a task your agent must finish, the tools it may use and the final state your team considers correct.

Talk to us →

CONTACT PRIVILEGE AI

Contact Us

Describe a task your agent must finish, the tools it may use and the final state your team considers correct.

When should we call?

By submitting, you agree that Privilege AI, a division of Tech1e LLC, may call or email you about this inquiry. Please leave confidential information out of your message.

Score the path,
not just the answer.

Privilege AI evaluates AI-generated work and the actions of AI agents, with authority set by measured evidence.