How to Evaluate AI Agents: Task Success, Tool Accuracy, Regression Tests, and Human Review

Evaluate an AI agent by what changed in the world, how it used tools, and whether the behavior remains reliable across repeated trials—not by whether its final message sounds convincing. Start with task cases that define an initial environment, user goal, permitted actions, and verifiable success conditions. Run the production agent and tool stack in an isolated fixture, record the full trace, then grade the final state, tool selection and arguments, safety invariants, latency, and cost.

Turn known-good cases and real failures into a regression suite that runs on prompt, model, policy, tool, and orchestration changes. Use deterministic checks where possible, model graders only for explicit subjective rubrics, and human review to calibrate graders and decide high-impact or ambiguous cases. A release passes only when critical invariants hold and any score change is large enough to distinguish from trial noise.

The AI agent evaluation stack in one table

This is broader than model benchmarking. The unit under test is the model plus agent harness, prompts, tools, memory, policies, and environment. A stronger model can still produce a worse product if a tool schema changed, a retry policy duplicates writes, or the context assembler retrieves stale state.

Define the evaluation contract before choosing metrics

Use precise names for the moving parts:

• A task is one scenario with inputs, an initial environment, permissions, and success criteria.
• A trial is one attempt at a task. The same task may need several trials because agent behavior is stochastic.
• A trace or trajectory is the ordered record of messages, model calls, tool proposals, policy decisions, executions, results, and state transitions.
• An outcome is the observable final environment state, such as an order becoming cancelled or a repository passing its tests.
• A grader applies one or more checks to the trace, output, or outcome.
• A suite is a versioned set of tasks designed to answer a particular question, such as capability or regression.

Anthropic's current guide to evals for AI agents uses the same crucial distinction between transcript and outcome. Google ADK's agent evaluation guidance likewise evaluates both final response quality and the trajectory of tool use. These are useful frameworks, but the contract should remain portable across vendors.

For the concise interview version of the broader evaluation system, start with how to build an LLM evaluation system. This article goes deeper on the agent-specific parts: environment state, tools, multi-step traces, retries, and human oversight.

Make the environment part of the fixture

An agent task is not reproducible if it depends on a live inbox, mutable web page, shared database, or another trial's leftovers. Each trial needs:

• a known snapshot or simulator state;
• isolated tenant, filesystem, queue, cache, and credential namespaces;
• pinned model, prompt, tool schema, policy, and fixture versions;
• deterministic clocks and random values where the task permits them;
• bounded network access or recorded dependencies; and
• cleanup that runs even after timeout or process failure.

Isolation protects validity. A later trial must not succeed because an earlier agent already created the file, warmed a cache, or left a useful note in memory.

Measure task success from the final state

The most important agent metric is task success rate: the proportion of trials whose required outcome and invariants pass.

task success rate = successful trials / total valid trials

The word “valid” matters. Infrastructure failures in the evaluation harness should be reported separately, not counted as agent failures or silently discarded. A high harness-error rate invalidates the comparison.

Prefer effect-level assertions

Suppose a support agent says, “I cancelled order A-100.” Grade the authoritative order state, not the sentence:

type CancellationFixture = {
  authenticatedUserId: string
  order: { id: string; status: string; cancelledBy?: string }
  ledger: { entries: Array<{ kind: string; orderId: string }> }
  notifications: Array<{ orderId: string; kind: string }>
}
 
function verifyCancellation(fixture: CancellationFixture) {
  const { authenticatedUserId, order, ledger, notifications } = fixture
  return [
    order.status === "cancelled",
    order.cancelledBy === authenticatedUserId,
    ledger.entries.filter(
      (entry) => entry.kind === "cancel" && entry.orderId === order.id
    ).length === 1,
    notifications.filter(
      (item) => item.kind === "cancelled" && item.orderId === order.id
    ).length === 1,
  ]
}

This catches false success messages, duplicate effects, cross-user mutations, and incomplete workflows. Coding agents should be graded with tests, static analysis, and repository state. Browser agents should be graded with backend or application state when possible, not the presence of a confirmation-looking screen.

Use binary gates and partial credit for different decisions

Binary success answers the release question: did the complete user task pass? Partial scores answer the diagnostic question: where did it fail?

Do not let partial credit turn a critical failure into a pass. An agent that identifies the right refund policy but refunds the wrong customer should fail the task and the safety gate, regardless of its weighted average.

Report slices, not only the global average

Break results down by customer tier, locale, tool, task family, risk class, input length, model route, and known failure mode. A one-point global improvement can hide a severe regression for a small but important slice.

Run the intended production mode too. If the user gets one attempt, pass@1 is the relevant experience; “at least one of five attempts succeeded” overstates reliability. Multiple trials are for estimating variance, not for quietly giving the deployed agent extra chances.

Score tool accuracy as more than name matching

Tool correctness has at least six dimensions:

Useful aggregate diagnostics include:

tool selection precision = expected or allowed calls / all attempted calls
tool selection recall    = required tool names observed / required tool names
argument accuracy        = required calls with valid arguments / required calls observed

Keep forbidden effects and authorization violations as counts with a target of zero. They must not disappear inside a 98% tool-accuracy score.

For security-focused cases, evaluate the real effect boundary: an unsafe proposal denied by policy is a successful defense, while a polite refusal after an unauthorized call is a failure. The AI agent security interview guide provides the corresponding abuse-case catalogue and authorization invariants.

Exact trajectories are sometimes too brittle

If two search tools can retrieve the same verified evidence, forcing one exact call sequence penalizes valid behavior. Prefer:

• required and forbidden calls;
• argument predicates rather than byte-for-byte JSON when fields are equivalent;
• partial-order constraints, such as verify_identity before refund.execute;
• state invariants and side-effect counts;
• maximum steps, cost, and latency; and
• trace rubrics for loops, recovery, or unnecessary work.

Require an exact sequence only when the sequence itself is the contract. Google ADK supports exact, in-order, and any-order trajectory matching; that choice is an evaluation decision, not a reason to default to exactness.

Build four suites instead of one overloaded benchmark

Capability suite: what can the agent do?

Use difficult tasks with headroom. A capability suite may have a modest pass rate because its purpose is to guide improvement and compare approaches. Refresh it when it saturates.

Regression suite: what must never break again?

Use stable, previously passing requirements and production failures with nearly deterministic grading. Run this suite on every material change. A failure should identify a behavior the team already promised to preserve.

Adversarial and safety suite: can controls contain misuse?

Include prompt injection, malicious tool output, cross-tenant IDs, approval replay, schema smuggling, excessive loops, stale permissions, and ambiguous timeouts. Test the model's proposal and the deterministic control that should deny or contain it.

Production-replay suite: does the lab still resemble reality?

Continuously sample consented, redacted, and policy-compliant production traces. Turn incidents, low-confidence outcomes, escalations, and reviewer disagreements into minimized fixtures. Preserve the failure mechanism without copying unnecessary personal data into an evaluation corpus.

Keep RAG component evaluation separate when retrieval is one stage of the agent. The production RAG system design shows how retrieval recall, reranking, groundedness, and answer quality isolate failures that a single end-to-end agent score would blur.

Turn evaluations into regression release gates

A candidate release is not just a prompt diff. Record at least:

• model provider, model identifier, routing policy, and inference settings;
• system prompt, templates, examples, and context-assembly version;
• agent harness, planner, memory, and recovery versions;
• tool catalog, schemas, adapters, and downstream fixture versions;
• authorization policy and approval logic versions; and
• dataset, grader, rubric, judge, and human-label versions.

Compare the candidate and baseline on the same tasks and environment snapshots. For nondeterministic behavior, run repeated trials and report uncertainty. A release policy might require:

• no critical safety or authorization failures;
• no regression on known incident cases;
• task-success change above a defined non-inferiority margin;
• no material degradation in priority slices;
• tool argument and ordering accuracy above fixed thresholds; and
• latency and cost within their budgets.

Do not tune the agent repeatedly on the only suite you report. Keep a development set for iteration and a holdout for release decisions. Review additions for duplicates and leakage, especially when production traces may already appear in prompt examples or retrieval data.

Pair the comparison at the task level

Running baseline and candidate on the same task fixtures removes some between-case variation. Inspect four groups:

Read the changed traces. A candidate “improvement” may be grader gaming; a “regression” may be a valid alternative path rejected by a brittle assertion.

Use human review as calibration and risk control

Human review is most valuable when selected deliberately. Route cases for review when:

• an action is high impact or irreversible;
• success depends on domain judgment rather than an objective state;
• deterministic and model graders disagree;
• the candidate and baseline swap winners;
• a slice drifts or a new failure cluster appears;
• the model grader is new or has changed version; or
• a small random sample is needed to detect blind spots.

Give reviewers an operational rubric

Bad prompt: “Rate this agent from 1 to 5.”

Better rubric:

Show the task, relevant state, and trace—not only the final answer. Blind the candidate identity and randomize pair order. Require a reason code and short evidence note, measure inter-reviewer agreement, and adjudicate important disagreements. Reviewers need a “task is ambiguous or broken” option so bad data does not become fake ground truth.

Calibrate model graders against humans

Model graders are useful for semantic quality that code cannot capture, but they are another versioned model component. On a human-labeled calibration set, measure false passes, false failures, agreement by slice, and stability after any judge or rubric change. Recheck samples regularly. Never let the same model family's confident stylistic preference become the sole judge of factual or policy correctness.

A runnable TypeScript agent evaluation harness

This dependency-free example evaluates a support agent across three order cases. It separates end-state task success from tool selection, arguments, ordering, forbidden effects, regression gates, and human-review routing. Replace the fake agent with your production adapter and the in-memory fixtures with isolated test environments.

type Json = null | boolean | number | string | Json[] | { [key: string]: Json }
 
type ToolCall = {
  name: string
  args: Record<string, Json>
  outcome: "ok" | "denied" | "error"
}
 
type AgentRun = {
  finalAnswer: string
  trace: ToolCall[]
  state: Record<string, Json>
  latencyMs: number
  costUsd: number
}
 
type ToolExpectation = {
  name: string
  args: Record<string, Json>
}
 
type EvalCase = {
  id: string
  prompt: string
  risk: "low" | "medium" | "high"
  requiredTools: ToolExpectation[]
  forbiddenTools: string[]
  mustOccurBefore: [string, string][]
  outcomeChecks: Array<{
    name: string
    check: (run: AgentRun) => boolean
  }>
}
 
interface AgentAdapter {
  run(testCase: EvalCase): Promise<AgentRun>
}
 
const cases: EvalCase[] = [
  {
    id: "cancel-eligible-order",
    prompt: "Cancel order A-100 because I changed my mind.",
    risk: "high",
    requiredTools: [
      { name: "orders.get", args: { orderId: "A-100" } },
      {
        name: "orders.cancel",
        args: { orderId: "A-100", reason: "customer_request" },
      },
    ],
    forbiddenTools: [],
    mustOccurBefore: [["orders.get", "orders.cancel"]],
    outcomeChecks: [
      {
        name: "order is cancelled",
        check: (run) => run.state.orderStatus === "cancelled",
      },
      {
        name: "one cancellation effect",
        check: (run) => run.state.cancelEffects === 1,
      },
    ],
  },
  {
    id: "do-not-cancel-shipped-order",
    prompt: "Cancel shipped order B-200.",
    risk: "medium",
    requiredTools: [
      { name: "orders.get", args: { orderId: "B-200" } },
    ],
    forbiddenTools: ["orders.cancel"],
    mustOccurBefore: [],
    outcomeChecks: [
      {
        name: "order remains shipped",
        check: (run) => run.state.orderStatus === "shipped",
      },
      {
        name: "agent explains the constraint",
        check: (run) => run.finalAnswer.includes("cannot be cancelled"),
      },
    ],
  },
  {
    id: "handle-missing-order",
    prompt: "Cancel order C-404.",
    risk: "low",
    requiredTools: [
      { name: "orders.get", args: { orderId: "C-404" } },
    ],
    forbiddenTools: ["orders.cancel"],
    mustOccurBefore: [],
    outcomeChecks: [
      {
        name: "no mutation occurred",
        check: (run) => run.state.cancelEffects === 0,
      },
      {
        name: "agent reports not found",
        check: (run) => run.finalAnswer.includes("could not find"),
      },
    ],
  },
]
 
function jsonEqual(left: Record<string, Json>, right: Record<string, Json>) {
  const normalize = (value: Json): Json => {
    if (Array.isArray(value)) return value.map(normalize)
    if (value && typeof value === "object") {
      return Object.fromEntries(
        Object.entries(value)
          .sort(([a], [b]) => a.localeCompare(b))
          .map(([key, child]) => [key, normalize(child)])
      )
    }
    return value
  }
  return JSON.stringify(normalize(left)) === JSON.stringify(normalize(right))
}
 
function gradeTools(testCase: EvalCase, run: AgentRun) {
  const requiredNamesFound = testCase.requiredTools.filter((expected) =>
    run.trace.some((call) => call.name === expected.name)
  ).length
 
  const exactRequiredFound = testCase.requiredTools.filter((expected) =>
    run.trace.some(
      (call) => call.name === expected.name && jsonEqual(call.args, expected.args)
    )
  ).length
 
  const expectedCalls = run.trace.filter((call) =>
    testCase.requiredTools.some((expected) => expected.name === call.name)
  ).length
 
  const forbiddenCalls = run.trace.filter((call) =>
    testCase.forbiddenTools.includes(call.name)
  )
 
  const orderingPassed = testCase.mustOccurBefore.every(([before, after]) => {
    const beforeIndex = run.trace.findIndex((call) => call.name === before)
    const afterIndex = run.trace.findIndex((call) => call.name === after)
    return beforeIndex >= 0 && afterIndex > beforeIndex
  })
 
  return {
    precision: run.trace.length === 0 ? 1 : expectedCalls / run.trace.length,
    recall:
      testCase.requiredTools.length === 0
        ? 1
        : requiredNamesFound / testCase.requiredTools.length,
    argumentAccuracy:
      requiredNamesFound === 0 ? 0 : exactRequiredFound / requiredNamesFound,
    forbiddenCalls: forbiddenCalls.length,
    orderingPassed,
  }
}
 
type CaseResult = {
  id: string
  taskPassed: boolean
  toolPrecision: number
  toolRecall: number
  argumentAccuracy: number
  safetyPassed: boolean
  orderingPassed: boolean
  latencyMs: number
  costUsd: number
  needsHumanReview: boolean
}
 
async function evaluate(agent: AgentAdapter): Promise<CaseResult[]> {
  return Promise.all(
    cases.map(async (testCase) => {
      const run = await agent.run(testCase)
      const tools = gradeTools(testCase, run)
      const taskPassed = testCase.outcomeChecks.every((item) => item.check(run))
      const safetyPassed = tools.forbiddenCalls === 0
 
      return {
        id: testCase.id,
        taskPassed,
        toolPrecision: tools.precision,
        toolRecall: tools.recall,
        argumentAccuracy: tools.argumentAccuracy,
        safetyPassed,
        orderingPassed: tools.orderingPassed,
        latencyMs: run.latencyMs,
        costUsd: run.costUsd,
        needsHumanReview:
          testCase.risk === "high" ||
          !taskPassed ||
          !safetyPassed ||
          !tools.orderingPassed,
      }
    })
  )
}
 
function average(values: number[]) {
  return values.reduce((sum, value) => sum + value, 0) / values.length
}
 
function summarize(results: CaseResult[]) {
  return {
    taskSuccess: average(results.map((item) => Number(item.taskPassed))),
    toolPrecision: average(results.map((item) => item.toolPrecision)),
    toolRecall: average(results.map((item) => item.toolRecall)),
    argumentAccuracy: average(results.map((item) => item.argumentAccuracy)),
    safetyFailures: results.filter((item) => !item.safetyPassed).length,
    reviewQueue: results.filter((item) => item.needsHumanReview).map((item) => item.id),
  }
}
 
function releaseGate(
  baseline: ReturnType<typeof summarize>,
  candidate: ReturnType<typeof summarize>
) {
  const failures: string[] = []
  if (candidate.safetyFailures > 0) failures.push("safety failure")
  if (candidate.taskSuccess < baseline.taskSuccess - 0.01) {
    failures.push("task success regressed by more than 1 percentage point")
  }
  if (candidate.argumentAccuracy < baseline.argumentAccuracy - 0.02) {
    failures.push("argument accuracy regressed by more than 2 percentage points")
  }
  return { passed: failures.length === 0, failures }
}
 
class FakeSupportAgent implements AgentAdapter {
  async run(testCase: EvalCase): Promise<AgentRun> {
    if (testCase.id === "cancel-eligible-order") {
      return {
        finalAnswer: "Order A-100 is cancelled.",
        trace: [
          { name: "orders.get", args: { orderId: "A-100" }, outcome: "ok" },
          {
            name: "orders.cancel",
            args: { orderId: "A-100", reason: "customer_request" },
            outcome: "ok",
          },
        ],
        state: { orderStatus: "cancelled", cancelEffects: 1 },
        latencyMs: 820,
        costUsd: 0.014,
      }
    }
 
    if (testCase.id === "do-not-cancel-shipped-order") {
      return {
        finalAnswer: "Order B-200 has shipped and cannot be cancelled.",
        trace: [
          { name: "orders.get", args: { orderId: "B-200" }, outcome: "ok" },
        ],
        state: { orderStatus: "shipped", cancelEffects: 0 },
        latencyMs: 510,
        costUsd: 0.009,
      }
    }
 
    return {
      finalAnswer: "I could not find order C-404.",
      trace: [
        { name: "orders.get", args: { orderId: "C-404" }, outcome: "error" },
      ],
      state: { cancelEffects: 0 },
      latencyMs: 430,
      costUsd: 0.008,
    }
  }
}
 
async function main() {
  const results = await evaluate(new FakeSupportAgent())
  const candidate = summarize(results)
  const baseline = {
    taskSuccess: 1,
    toolPrecision: 1,
    toolRecall: 1,
    argumentAccuracy: 1,
    safetyFailures: 0,
    reviewQueue: [] as string[],
  }
  const gate = releaseGate(baseline, candidate)
 
  console.log(`task success: ${candidate.taskSuccess * 100}%`)
  console.log(`tool precision: ${candidate.toolPrecision * 100}%`)
  console.log(`tool recall: ${candidate.toolRecall * 100}%`)
  console.log(`argument accuracy: ${candidate.argumentAccuracy * 100}%`)
  console.log(`human review: ${candidate.reviewQueue.join(", ")}`)
  console.log(`release gate: ${gate.passed ? "PASS" : "FAIL"}`)
}
 
void main()

Expected output:

task success: 100%
tool precision: 100%
tool recall: 100%
argument accuracy: 100%
human review: cancel-eligible-order
release gate: PASS

The thresholds are examples, not universal recommendations. Choose margins from product risk, baseline variance, traffic, and the smallest change worth blocking. For a real stochastic agent, run several trials per case, retain every trial rather than only the best, and compute confidence intervals or an appropriate paired test before interpreting a small difference.

The code deliberately queues the successful high-risk cancellation for human review. Automated success and a review requirement can both be true: one measures behavior, while the other enforces an oversight policy.

Connect offline evals to production evidence

Offline evaluation protects known behavior before users see a release. Production monitoring finds distribution shifts and unknown failures. Connect them with the same identifiers:

release → run → task → model call → tool proposal → policy decision
        → tool execution → outcome → user feedback → review label

Record model, prompt, tool, policy, tenant-safe slice, timing, token usage, cost, errors, and terminal outcome. Tool arguments and results often contain sensitive data, so default to hashes, schemas, classifications, and redacted samples rather than unrestricted payload logging. OpenTelemetry's current GenAI semantic attributes cover agent and tool operations while explicitly warning that arguments and results can be sensitive.

Useful online signals include successful outcome rate, escalation rate, undo or compensation rate, repeated tool calls, denied proposals, abandonment, latency, cost per successful task, and delayed business truth. User ratings are helpful signals, but they are sparse and self-selected; they do not replace outcome verification.

Sample traces using documented privacy and retention rules. Prioritize incidents, denials, high cost, unusual paths, changed slices, and random coverage. Every confirmed failure should produce three artifacts: a root-cause classification, a minimal reproducible task, and an owner for the fix and permanent regression case.

Common evaluation-system failure modes

Evaluation can fail too. Maintain the harness, fixtures, graders, labels, and review process like production software, with owners, change review, observability, and tests of the tests.

Production checklist

Task and dataset design

• Every case defines initial state, identity, permissions, goal, allowed variability, and verifiable success conditions.
• Capability, regression, adversarial, and production-replay suites have separate purposes and reporting.
• Priority user, language, risk, tool, and failure-mode slices have explicit coverage.
• Development and release holdout sets are separated, deduplicated, and checked for leakage.
• Production examples are consented or otherwise lawfully handled, minimized, redacted, access-controlled, and retained by policy.

Harness and graders

• Trials start from isolated, versioned environment snapshots and always clean up.
• Harness failures are reported separately from agent failures.
• End-state and deterministic graders are preferred where the truth is machine-verifiable.
• Tool grading covers selection, semantic arguments, ordering, authorization, execution effects, and result interpretation.
• Model graders use explicit rubrics and are calibrated against blinded human labels by slice.
• Full traces remain inspectable with secret and personal-data redaction.

Release and operations

• Candidate and baseline run on paired cases with repeated trials where behavior is stochastic.
• Critical safety, authorization, and duplicate-effect failures cannot be averaged away.
• Task success, tool accuracy, cost, latency, and priority slices have named release thresholds and owners.
• Model, prompt, harness, tools, policy, dataset, graders, and judge versions are recorded for every run.
• Canary, rollback, and kill-switch paths are tested before increasing agent autonomy.
• Production incidents and adjudicated review failures become regression fixtures.

How to explain this in a system design interview

Use an outcome-first answer:

I would evaluate the complete agent system, not only the model's final text. Each versioned task starts from an isolated environment and defines a user goal, identity, allowed tools, risk, final-state assertions, and safety invariants. The harness records the full trajectory. Deterministic graders verify end state and tool effects; trace graders score selection, arguments, ordering, budgets, and recovery; calibrated rubrics handle subjective interaction quality. I run repeated paired trials against a pinned baseline, gate critical slices and safety failures separately, deploy through a canary, and feed incidents plus human-adjudicated disagreements back into regression cases.

Then make one trade-off explicit: exact trajectories improve safety checks when order matters, but they are brittle for open-ended tasks, so grade outcomes and partial-order constraints by default.

Realistic follow-up questions

How would you evaluate an agent whose result is subjective?

Decompose objective parts first: required sources, policy compliance, tool effects, omissions, and factual checks. Use an anchored rubric for the remaining quality dimensions, calibrate a model grader against expert labels, and retain human adjudication for low-agreement or high-impact cases.

How do you know whether a one-point improvement is real?

Run baseline and candidate on the same fixtures, repeat stochastic trials, inspect paired wins and losses, and report a confidence interval or suitable paired statistical test. The required evidence depends on variance and the smallest effect worth shipping; one run per case cannot establish a small gain.

Would you grade the exact tool trajectory?

Only where order is a contract. Otherwise I grade required and forbidden calls, semantic argument predicates, partial-order constraints, state invariants, and budgets so alternative valid strategies pass.

How would you evaluate failure recovery?

Inject timeouts before and after side effects, duplicate delivery, stale memory, policy denial, cancellation, expired approval, and process restart. Verify the agent neither claims false success nor duplicates effects and that the workflow converges to the correct terminal state. The multi-agent system design guide shows the durable recovery architecture those tests exercise.

When is human review mandatory?

When policy or risk requires approval, when truth is genuinely subjective, when automated graders disagree, and when calibrating or auditing those graders. Risk tiering should decide the review path, not model confidence alone.

The Agentic Systems & Tool Use assessment tests these architecture choices directly. For a fuller Staff-level control model, continue with the Bounded Autonomy course lesson.

The release decision this system should enable

A useful evaluation system turns “the agent feels better” into a traceable decision:

• what improved, regressed, or remained uncertain;
• which users, tasks, tools, and risk slices changed;
• whether observed effects are larger than trial and harness noise;
• which critical failures block release regardless of the average; and
• what production evidence will confirm or overturn the offline result.

That is the standard to aim for. A benchmark score describes a run. A production evaluation system protects a promise to users and creates the evidence required to change the agent safely.