Evaluating Agents and Benchmarks
What you’ll learn. Why agent evaluation differs from model evaluation; outcome versus process metrics; the main benchmark families (SWE-bench-style, tau-bench-style, WebArena/OSWorld-style, research and tool-use suites) and what they test; the pitfalls (flakiness, contamination, grader bugs, variance, reward hacking); and a practical recipe for building an eval for your own agent. General LLM evaluation is in Evaluation.
Why agent evaluation is harder
Section titled “Why agent evaluation is harder”A single-turn model output can be compared with a reference. An agent produces a trajectory in a stateful, often nondeterministic environment. Complications:
- Success is defined by the final state or side effects, not the text (was the right row updated? did tests pass?).
- Many valid paths to success; step-by-step reference matching penalizes legitimate alternatives.
- Stochasticity in the model, tools, websites and simulated users: the same task can pass on one run and fail on the next.
- Cost and time: one run may take minutes and dollars, limiting sample sizes.
- Environment setup (containers, databases, accounts) is real engineering and a source of spurious failures.
What to measure
Section titled “What to measure”| Level | Metric | Notes |
|---|---|---|
| Outcome | Task success rate (resolved / completed) | The headline number |
| Reliability | pass^k: succeeds on all attempts | Measures consistency |
| Capability | pass@k: succeeds on any of | Measures upper reach (see Test-time compute) |
| Efficiency | Steps, tokens, wall-clock, USD per task | Cost-adjusted success |
| Process | Tool-call validity, error recovery, policy adherence | Diagnostic |
| Safety | Unsafe actions, unauthorized side effects | Often more important than success |
The distinction between pass@k and pass^k matters. If a single-run success probability is (per task), then pass^k . An agent with 80% per-run success has pass^8 of about 17%: customer-facing reliability collapses quickly. Tau-bench popularized pass^k precisely because deployments need consistency.
Benchmark families
Section titled “Benchmark families”SWE-bench style (software engineering)
Section titled “SWE-bench style (software engineering)”SWE-bench (Jimenez et al., 2023) uses real GitHub issues from popular Python repositories. The agent receives the repository at the issue’s commit and the issue text, and must produce a patch. Success: previously failing tests now pass and previously passing tests still pass, run in a container. Variants: SWE-bench Verified (a subset of about 500 instances human-validated by OpenAI-affiliated annotators to be well-specified and solvable), SWE-bench Lite, and later multilingual and harder variants (e.g. Multi-SWE-bench, SWE-bench Pro, and live/contamination-resistant versions). Resolve rates on Verified rose from low single digits in 2023 to a large majority of tasks by 2025-2026 for frontier agents, which is the usual benchmark lifecycle: saturation within about two years, so check the current frontier yourself.
What it tests: localization, multi-file edits, test-driven iteration. What it misses: ambiguity handling, long features, code quality and maintainability, non-Python ecosystems, and the fact that issue-to-patch is a narrow slice of engineering.
Tau-bench style (tool-agent-user interaction under policy)
Section titled “Tau-bench style (tool-agent-user interaction under policy)”tau-bench (Yao et al., 2024) simulates customer-service domains (retail, airline). An LLM-simulated user chats with the agent; the agent has database tools and must follow a written policy document (e.g. refund rules). Success is determined by comparing the final database state to the goal state. It reports pass^k, and found that even strong function-calling agents were inconsistent across repeated trials of the same task. It tests policy adherence, multi-turn dialogue and tool use together. Weaknesses: simulated-user noise and limited domains; follow-ups extended it to more domains and to dual-control settings where the user also acts.
WebArena / OSWorld style (GUI agents)
Section titled “WebArena / OSWorld style (GUI agents)”Self-hosted realistic web apps or full operating systems with execution-based checkers (Computer-use agents). Reliability and latency dominate. Known issues: brittle checkers (answer-format sensitivity), tasks whose ground truth changed, environments that drift, and heavy infrastructure.
Others worth knowing
Section titled “Others worth knowing”- GAIA: general assistant questions requiring tools and browsing with unambiguous short answers.
- BrowseComp: hard-to-find facts needing persistent browsing.
- AgentBench, ToolBench, BFCL (Berkeley Function Calling Leaderboard): function-calling and multi-step tool use.
- MLE-bench, PaperBench, RE-Bench: research and machine-learning engineering tasks with long horizons.
- Terminal-oriented benchmarks for shell agents.
- METR time-horizon analysis: measures the length of tasks (in human-expert time) that agents complete at 50% success; they reported this horizon doubling roughly every several months over recent years. This is a useful complement to fixed-set accuracy.
Names and versions change quickly; treat this list as a map, not a leaderboard.
Pitfalls
Section titled “Pitfalls”Variance. With 100 tasks and 60% success, the standard error of the mean is percentage points, so a 3-point improvement is noise. Use the same task set, run multiple seeds, report confidence intervals, and use paired comparisons (McNemar or paired bootstrap) between agents on the same tasks.
import numpy as np
def paired_bootstrap(a, b, iters=10_000, seed=0): """a, b: 0/1 arrays of per-task success for two agents on the same tasks.""" rng = np.random.default_rng(seed) a, b = np.asarray(a), np.asarray(b) n = len(a) diffs = [(a[i].mean() - b[i].mean()) for i in (rng.integers(0, n, n) for _ in range(iters))] lo, hi = np.percentile(diffs, [2.5, 97.5]) return a.mean() - b.mean(), (lo, hi)Contamination. Benchmarks leak into pretraining and fine-tuning data; scores overstate ability. Mitigations: held-out private sets, time-split tasks created after the training cutoff, dynamic benchmarks that regenerate.
Grader and task bugs. A meaningful share of tasks in many benchmarks have flawed tests, ambiguous specs or unreachable goals; this is why curated “Verified” subsets exist and why leaderboards get corrected. Read failed trajectories; if every strong agent fails the same task, suspect the task.
Reward hacking at eval time. Agents can read the benchmark’s reference solution from git history or the file system, or edit tests. Isolate the grader and scrub leakage (e.g. future commits).
Scaffold confound. A benchmark score is for model + harness + prompts + budget. Comparing models means holding the scaffold fixed; comparing across leaderboard entries often does not. Record step limits, token budget, number of attempts and whether a verifier selected among samples.
Benchmark-reality gap. High benchmark scores may not translate to your workflow. Public benchmarks are sanity checks, not substitutes for your own tasks.
Metric gaming through cost. Report cost per resolved task, not accuracy alone. An agent that spends USD 40 per task to gain 2 points may be uneconomic.
LLM judges for trajectories
Section titled “LLM judges for trajectories”For tasks without programmatic checks, a model can grade final outputs or whole transcripts against a rubric. Rules: use a rubric with concrete criteria, grade with a model at least as capable as the agent, randomize ordering for comparisons, calibrate against a sample of human grades (report agreement), and keep judge prompts fixed across experiments. See Self-refinement and feedback for judge biases.
Building your own eval
Section titled “Building your own eval”A practical recipe:
- Collect real tasks from logs, tickets, or your own work. Start with 20-50; quality over quantity.
- Define success precisely and make it programmatic when possible (state checks, tests, exact-match on key fields); fall back to rubric-based LLM judging with human spot checks.
- Make environments reproducible: containers, seeded databases, mocked external services, frozen clocks.
- Include negative and safety cases: tasks where the right behavior is to refuse, ask a question, or do nothing.
- Run trials per task; track pass@1, pass^k, cost.
- Read transcripts, especially failures, and label failure categories (wrong tool, bad plan, ignored instruction, environment error). This is the highest-yield activity in agent development.
- Use it as a regression suite in CI: every prompt, tool or model change must not lower the score.
- Refresh periodically so the eval does not become a target you overfit.
from dataclasses import dataclassfrom typing import Callable
@dataclassclass Task: id: str prompt: str setup: Callable[[], object] # builds a fresh environment check: Callable[[object, str], bool] # (env, final_answer) -> success
def evaluate(agent, tasks, k=3): rows = [] for t in tasks: results = [] for _ in range(k): env = t.setup() answer = agent(t.prompt, env) results.append(t.check(env, answer)) rows.append({"id": t.id, "pass1": sum(results)/k, "pass_all": all(results), "pass_any": any(results)}) return rowsOffline evals versus online monitoring
Section titled “Offline evals versus online monitoring”Offline evals gate releases; production traces reveal what users actually do. Mine failures from production (thumbs-down, escalations, tool errors) into new eval tasks. See Production and observability.
Key takeaways
Section titled “Key takeaways”- Agent success is measured on end state over trajectories; report pass@1, pass^k for consistency, and cost.
- SWE-bench, tau-bench and WebArena/OSWorld cover code, policy-bound tool dialogue and GUI use; benchmarks saturate and drift, so verify current state.
- Variance, contamination, flawed graders, scaffold differences and hackable environments commonly invalidate comparisons.
- Build your own small, reproducible, programmatic eval from real tasks and read failed transcripts.
Exercises
Section titled “Exercises”- Compute. An agent succeeds 85% per run on each task independently. What is pass^1, pass^4 and pass^10?
- Compute. Two agents score 62% and 66% on the same 150 tasks. Is that difference likely significant? What additional data do you need?
- Conceptual. Why is reporting SWE-bench scores without stating the number of attempts and selection method misleading?
- Hands-on. Build a 10-task evaluation for a file-manipulation agent using temporary directories and a
checkfunction, and run it with theevaluatefunction.
Solutions (1, 2)
- , , .
- Standard error for a single proportion near 0.64 with is about 3.9 points, so a 4-point difference is within noise unless paired per-task data show consistent wins; run the paired bootstrap and more seeds.
Further reading
Section titled “Further reading”- Jimenez et al., “SWE-bench: Can Language Models Resolve Real-World GitHub Issues?” (2023).
- Yao et al., “tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains” (2024).
- Zhou et al., “WebArena” (2023); Xie et al., “OSWorld” (2024); Mialon et al., “GAIA” (2023).
- Kapoor et al., “AI Agents That Matter” (2024), on cost-aware evaluation and benchmark hygiene.
- Kwa et al., “Measuring AI Ability to Complete Long Tasks” (METR, 2025).
- Zheng et al., “Judging LLM-as-a-Judge” (2023); see also Evaluation.