Evaluation skillsHow-to

How to Evaluate AI Agents and Multi-Step Tasks

Inspect the result, the tool-use record and the constraints an agent had to respect. A confident final message is only one piece of evidence.

4 min read · Published September 9, 2026

You'll learn

  1. Check the final environment state or artifact, not only the agent’s claim.
  2. Separate outcome quality from permission, process and tool-use compliance.
  3. Distinguish agent failures from missing inputs or broken evaluation environments.
  4. Record enough evidence to reproduce the judgment.
In this guide
  1. Inspect the task, trace and result together
  2. The message and the outcome can disagree
  3. Do not blame the model for the wrong failure
  4. Judge recovery in context

An agent can take several actions before answering: search, open files, call tools, edit a document or change an application. Review must therefore ask two questions: did the requested result exist, and did the agent reach it within the permitted scope?

Quick definitionTrajectory
The available record of an agent’s steps, including tool calls, returned results and messages. Review the observable record rather than inventing hidden reasoning.

Inspect the task, trace and result together

An agent review

  1. Establish the starting conditions

    Read the user request, permissions, supplied files and initial environment. Identify what success and prohibited actions mean for this task.

  2. Inspect the final artifact or state

    Confirm the requested file, change or result exists and contains the required content.

  3. Trace consequential actions

    Check the tool calls that support the result and any action that crossed a permission boundary.

  4. Classify the failure source

    Separate a bad agent decision from a tool outage, missing fixture or ambiguous test assumption.

  5. Score outcome and process

    Use the rubric’s dimensions. A good result does not automatically excuse an unauthorized action.

  6. Record reproducible evidence

    Identify the artifact, trace step and observable mismatch without claiming access to unavailable state.

The message and the outcome can disagree

Evaluation example

Prompt
Illustrative task. Create a draft meeting agenda in the provided workspace. Do not send it. Both agents write a complete agenda file. Agent A then emails it; Agent B leaves it as a draft.
Response A
The agenda is ready, and I sent it to the team so everyone can prepare.
Response B
The draft agenda is saved in the workspace for review.
Decision
Response B
Why
B completes the request within scope. A creates the artifact but violates the explicit instruction not to send it. The evidence is the successful email action in the trace, not simply A’s wording. Outcome and permission compliance should be recorded separately where the rubric supports that distinction.

Do not blame the model for the wrong failure

If a tool returns an access error, examine what the agent does next. It may correctly explain the limitation, or falsely claim completion. The outage and the false completion claim are different events. If a test assumes a file location the prompt never specified, that may be a test-design problem rather than a task failure.

Anthropic’s engineering guidance on agent evaluations distinguishes the transcript from the outcome and recommends inspecting both. This matters because a final message can sound successful while the requested state change never occurred.

Common mistake

Scoring only the final answer

A message cannot prove that a file opens, a reservation exists or a tool action respected the user’s limits. Inspect the observable artifact and trace supplied by the evaluation. If either is unavailable, record the resulting limit.

Judge recovery in context

Repeated tool calls may waste time, but a second attempt after a transient failure can be reasonable. Check whether the agent used new evidence, respected limits and stopped when further action was unwarranted. Do not penalize every detour equally or reward a short trace that simply skipped required verification.

Before you finalize the review

  • Verify the actual requested result.
  • Check the trace for prohibited side effects.
  • Locate evidence for any completion claim.
  • Separate environment failures from agent choices.
  • Use the supplied rubric for efficiency and recovery.
  • Record uncertainty when the trace or final state is incomplete.

Key takeaway

Evaluate what changed, what the agent did and what it could legitimately claim from the evidence.