Skip to content

Evaluating Agents and Benchmarks

What you’ll learn. Why agent evaluation differs from model evaluation; outcome versus process metrics; the main benchmark families (SWE-bench-style, tau-bench-style, WebArena/OSWorld-style, research and tool-use suites) and what they test; the pitfalls (flakiness, contamination, grader bugs, variance, reward hacking); and a practical recipe for building an eval for your own agent. General LLM evaluation is in Evaluation.

A single-turn model output can be compared with a reference. An agent produces a trajectory in a stateful, often nondeterministic environment. Complications:

  • Success is defined by the final state or side effects, not the text (was the right row updated? did tests pass?).
  • Many valid paths to success; step-by-step reference matching penalizes legitimate alternatives.
  • Stochasticity in the model, tools, websites and simulated users: the same task can pass on one run and fail on the next.
  • Cost and time: one run may take minutes and dollars, limiting sample sizes.
  • Environment setup (containers, databases, accounts) is real engineering and a source of spurious failures.
Level Metric Notes
Outcome Task success rate (resolved / completed) The headline number
Reliability pass^k: succeeds on all kk attempts Measures consistency
Capability pass@k: succeeds on any of kk Measures upper reach (see Test-time compute)
Efficiency Steps, tokens, wall-clock, USD per task Cost-adjusted success
Process Tool-call validity, error recovery, policy adherence Diagnostic
Safety Unsafe actions, unauthorized side effects Often more important than success

The distinction between pass@k and pass^k matters. If a single-run success probability is pp (per task), then pass^k =pk=p^k. An agent with 80% per-run success has pass^8 of about 17%: customer-facing reliability collapses quickly. Tau-bench popularized pass^k precisely because deployments need consistency.

SWE-bench (Jimenez et al., 2023) uses real GitHub issues from popular Python repositories. The agent receives the repository at the issue’s commit and the issue text, and must produce a patch. Success: previously failing tests now pass and previously passing tests still pass, run in a container. Variants: SWE-bench Verified (a subset of about 500 instances human-validated by OpenAI-affiliated annotators to be well-specified and solvable), SWE-bench Lite, and later multilingual and harder variants (e.g. Multi-SWE-bench, SWE-bench Pro, and live/contamination-resistant versions). Resolve rates on Verified rose from low single digits in 2023 to a large majority of tasks by 2025-2026 for frontier agents, which is the usual benchmark lifecycle: saturation within about two years, so check the current frontier yourself.

What it tests: localization, multi-file edits, test-driven iteration. What it misses: ambiguity handling, long features, code quality and maintainability, non-Python ecosystems, and the fact that issue-to-patch is a narrow slice of engineering.

Tau-bench style (tool-agent-user interaction under policy)

Section titled “Tau-bench style (tool-agent-user interaction under policy)”

tau-bench (Yao et al., 2024) simulates customer-service domains (retail, airline). An LLM-simulated user chats with the agent; the agent has database tools and must follow a written policy document (e.g. refund rules). Success is determined by comparing the final database state to the goal state. It reports pass^k, and found that even strong function-calling agents were inconsistent across repeated trials of the same task. It tests policy adherence, multi-turn dialogue and tool use together. Weaknesses: simulated-user noise and limited domains; follow-ups extended it to more domains and to dual-control settings where the user also acts.

Self-hosted realistic web apps or full operating systems with execution-based checkers (Computer-use agents). Reliability and latency dominate. Known issues: brittle checkers (answer-format sensitivity), tasks whose ground truth changed, environments that drift, and heavy infrastructure.

  • GAIA: general assistant questions requiring tools and browsing with unambiguous short answers.
  • BrowseComp: hard-to-find facts needing persistent browsing.
  • AgentBench, ToolBench, BFCL (Berkeley Function Calling Leaderboard): function-calling and multi-step tool use.
  • MLE-bench, PaperBench, RE-Bench: research and machine-learning engineering tasks with long horizons.
  • Terminal-oriented benchmarks for shell agents.
  • METR time-horizon analysis: measures the length of tasks (in human-expert time) that agents complete at 50% success; they reported this horizon doubling roughly every several months over recent years. This is a useful complement to fixed-set accuracy.

Names and versions change quickly; treat this list as a map, not a leaderboard.

Variance. With 100 tasks and 60% success, the standard error of the mean is 0.6⋅0.4/100≈4.9\sqrt{0.6\cdot0.4/100}\approx 4.9 percentage points, so a 3-point improvement is noise. Use the same task set, run multiple seeds, report confidence intervals, and use paired comparisons (McNemar or paired bootstrap) between agents on the same tasks.

import numpy as np
def paired_bootstrap(a, b, iters=10_000, seed=0):
"""a, b: 0/1 arrays of per-task success for two agents on the same tasks."""
rng = np.random.default_rng(seed)
a, b = np.asarray(a), np.asarray(b)
n = len(a)
diffs = [(a[i].mean() - b[i].mean())
for i in (rng.integers(0, n, n) for _ in range(iters))]
lo, hi = np.percentile(diffs, [2.5, 97.5])
return a.mean() - b.mean(), (lo, hi)

Contamination. Benchmarks leak into pretraining and fine-tuning data; scores overstate ability. Mitigations: held-out private sets, time-split tasks created after the training cutoff, dynamic benchmarks that regenerate.

Grader and task bugs. A meaningful share of tasks in many benchmarks have flawed tests, ambiguous specs or unreachable goals; this is why curated “Verified” subsets exist and why leaderboards get corrected. Read failed trajectories; if every strong agent fails the same task, suspect the task.

Reward hacking at eval time. Agents can read the benchmark’s reference solution from git history or the file system, or edit tests. Isolate the grader and scrub leakage (e.g. future commits).

Scaffold confound. A benchmark score is for model + harness + prompts + budget. Comparing models means holding the scaffold fixed; comparing across leaderboard entries often does not. Record step limits, token budget, number of attempts and whether a verifier selected among samples.

Benchmark-reality gap. High benchmark scores may not translate to your workflow. Public benchmarks are sanity checks, not substitutes for your own tasks.

Metric gaming through cost. Report cost per resolved task, not accuracy alone. An agent that spends USD 40 per task to gain 2 points may be uneconomic.

For tasks without programmatic checks, a model can grade final outputs or whole transcripts against a rubric. Rules: use a rubric with concrete criteria, grade with a model at least as capable as the agent, randomize ordering for comparisons, calibrate against a sample of human grades (report agreement), and keep judge prompts fixed across experiments. See Self-refinement and feedback for judge biases.

A practical recipe:

  1. Collect real tasks from logs, tickets, or your own work. Start with 20-50; quality over quantity.
  2. Define success precisely and make it programmatic when possible (state checks, tests, exact-match on key fields); fall back to rubric-based LLM judging with human spot checks.
  3. Make environments reproducible: containers, seeded databases, mocked external services, frozen clocks.
  4. Include negative and safety cases: tasks where the right behavior is to refuse, ask a question, or do nothing.
  5. Run k≥3k \ge 3 trials per task; track pass@1, pass^k, cost.
  6. Read transcripts, especially failures, and label failure categories (wrong tool, bad plan, ignored instruction, environment error). This is the highest-yield activity in agent development.
  7. Use it as a regression suite in CI: every prompt, tool or model change must not lower the score.
  8. Refresh periodically so the eval does not become a target you overfit.
from dataclasses import dataclass
from typing import Callable
@dataclass
class Task:
id: str
prompt: str
setup: Callable[[], object] # builds a fresh environment
check: Callable[[object, str], bool] # (env, final_answer) -> success
def evaluate(agent, tasks, k=3):
rows = []
for t in tasks:
results = []
for _ in range(k):
env = t.setup()
answer = agent(t.prompt, env)
results.append(t.check(env, answer))
rows.append({"id": t.id, "pass1": sum(results)/k,
"pass_all": all(results), "pass_any": any(results)})
return rows

Offline evals gate releases; production traces reveal what users actually do. Mine failures from production (thumbs-down, escalations, tool errors) into new eval tasks. See Production and observability.

  • Agent success is measured on end state over trajectories; report pass@1, pass^k for consistency, and cost.
  • SWE-bench, tau-bench and WebArena/OSWorld cover code, policy-bound tool dialogue and GUI use; benchmarks saturate and drift, so verify current state.
  • Variance, contamination, flawed graders, scaffold differences and hackable environments commonly invalidate comparisons.
  • Build your own small, reproducible, programmatic eval from real tasks and read failed transcripts.
  1. Compute. An agent succeeds 85% per run on each task independently. What is pass^1, pass^4 and pass^10?
  2. Compute. Two agents score 62% and 66% on the same 150 tasks. Is that difference likely significant? What additional data do you need?
  3. Conceptual. Why is reporting SWE-bench scores without stating the number of attempts and selection method misleading?
  4. Hands-on. Build a 10-task evaluation for a file-manipulation agent using temporary directories and a check function, and run it with the evaluate function.
Solutions (1, 2)
  1. 0.850.85, 0.854≈0.520.85^4\approx0.52, 0.8510≈0.200.85^{10}\approx0.20.
  2. Standard error for a single proportion near 0.64 with n=150n=150 is about 3.9 points, so a 4-point difference is within noise unless paired per-task data show consistent wins; run the paired bootstrap and more seeds.
  • Jimenez et al., “SWE-bench: Can Language Models Resolve Real-World GitHub Issues?” (2023).
  • Yao et al., “tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains” (2024).
  • Zhou et al., “WebArena” (2023); Xie et al., “OSWorld” (2024); Mialon et al., “GAIA” (2023).
  • Kapoor et al., “AI Agents That Matter” (2024), on cost-aware evaluation and benchmark hygiene.
  • Kwa et al., “Measuring AI Ability to Complete Long Tasks” (METR, 2025).
  • Zheng et al., “Judging LLM-as-a-Judge” (2023); see also Evaluation.