Skip to content

Evaluating LLMs

What you’ll learn. Why evaluation is the hardest and most valuable part of LLM engineering; how public benchmarks work and fail (saturation, contamination, Goodhart effects); automatic metrics versus LLM-as-judge and its biases; how to run human evaluation and measure agreement; how to build task-specific evals and use them for eval-driven development; statistics for small samples and nondeterminism; and red-teaming for safety and security.

Classical ML has a held-out test set and one metric. LLM outputs are open-ended text, tasks are diverse, behavior is sensitive to prompts, and models are stochastic. Common failure patterns:

  • Teams ship on “vibes” from a handful of examples, then regress silently when swapping a model or prompt.
  • Leaderboard rank doesn’t predict performance on your task.
  • The metric measures what is easy (string match, length) rather than what matters.

A reliable mental model: an eval is a test suite for a probabilistic component. Its value comes from being representative of production, cheap enough to run on every change, and trusted.

Benchmarks provide a shared yardstick for comparing models. Categories (specific names turn over quickly; the categories persist):

Capability Examples of benchmark style
Knowledge and broad reasoning multiple-choice exams across subjects (MMLU-style, and harder successors such as MMLU-Pro and GPQA-style expert questions)
Math grade-school word problems (GSM8K), competition problems (MATH, AIME-style), newer proof/olympiad sets
Code function synthesis with unit tests (HumanEval, MBPP), repository-level issue fixing (SWE-bench-style), live competitive programming
Instruction following verifiable-constraint prompts (IFEval-style), preference arenas
Long context retrieval and multi-hop over long inputs (needle-in-haystack and harder variants)
Agents / tools multi-turn tool use in simulated domains (tau-bench-style), web and computer-use tasks
Safety harmful-request refusal sets, jailbreak suites, bias and toxicity probes
Truthfulness misconception and hallucination probes
  1. Saturation. Once models score 90%+, differences are within noise or label error (several popular benchmarks contain a noticeable fraction of mislabeled items), so the field moves to harder successors.
  2. Contamination. If benchmark items appeared in pretraining or post-training data, scores overstate ability. It happens through scraping of the benchmark itself, near-duplicates, solutions on forums, and inadvertently through synthetic data. Detection: n-gram overlap checks, canary strings, perplexity-based memorization tests, comparing performance on the original versus a freshly rewritten or newly collected version of the benchmark. Mitigation: dynamic or time-split benchmarks (problems released after the model’s data cutoff, e.g. rolling competitive-programming sets), private held-out sets, and rewriting/perturbing items.
  3. Prompt and harness sensitivity. Few-shot count, template, answer extraction regex, sampling settings and scoring (log-likelihood of options vs generated answer) can change scores by several points. Numbers from different labs’ harnesses aren’t comparable; use a single open harness (e.g. EleutherAI’s lm-evaluation-harness, HELM, or similar) for apples-to-apples.
  4. Goodhart’s law. Optimizing for a benchmark (directly or through data selection) decouples score from the underlying ability.
  5. Narrow construct validity. Multiple-choice knowledge isn’t the same as being a good assistant, coder or agent.
  6. Preference arenas (crowdsourced pairwise comparisons aggregated with Bradley-Terry/Elo-style ratings) capture human taste on real prompts but reward style, length and formatting; style-controlled variants try to adjust for this, and the population of voters may not match your users.
  7. Single numbers hide variance. Report confidence intervals, number of runs and the settings (temperature, reasoning effort, tool access).

Treat public benchmarks as a coarse filter for candidate models, never as a substitute for task-specific evaluation. Model rankings change within months; build your process to be model-agnostic.

  • Exact match / accuracy / F1 for closed-form answers (classification, extraction, QA with short answers). Normalize case, punctuation, units; for numbers parse rather than string-match.
  • Execution-based: run unit tests for code (pass@k, the probability at least one of kk samples passes; use the unbiased estimator from Chen et al., 2021: pass@k=1−(n−ck)/(nk)\text{pass@}k = 1-\binom{n-c}{k}/\binom{n}{k} with nn samples, cc correct). SQL results compared to gold; schema validation for structured output.
  • Programmatic checks: format constraints, length, presence of citations, forbidden content, regex.
  • Overlap metrics (BLEU, ROUGE) and embedding similarity (BERTScore): cheap but correlate weakly with quality for open-ended generation; fine for narrow tasks like translation or extractive summaries with good references.
  • Perplexity / bits-per-byte for base-model comparison only (NLP Foundations).
  • Retrieval metrics for RAG: recall@k, MRR, nDCG (RAG).

Prefer verifiable checks whenever the task allows: they’re cheap, deterministic and unbiased.

For open-ended quality (helpfulness, tone, faithfulness, reasoning quality), use a strong LLM to grade outputs. It scales to thousands of cases at low cost, and with careful design agrees with human raters about as much as humans agree with each other on many tasks (Zheng et al., 2023).

Modes:

  • Pointwise with a rubric: score 1-5 or pass/fail per criterion.
  • Pairwise: which of A or B is better (more reliable than absolute scores).
  • Reference-based: compare to a gold answer (“does the response contain the same facts as the reference?”).
  • Reference-free with checklists: decompose the question into binary checks.

Known biases:

  • Position bias: prefers the first (or second) answer. Fix: evaluate both orders and count only consistent verdicts.
  • Verbosity/length bias: prefers longer answers. Fix: instruct and control for length; compare length-matched.
  • Self-preference: favors outputs from its own model family. Fix: use a different family as judge, or ensembles.
  • Style over substance, sycophancy to confident tone; weak at verifying factual correctness or hard math/code without tools or reference.
  • Score compression: 7/10 for everything. Use anchored rubrics with examples per level, or binary decisions.

Practices:

JUDGE = """You are grading an answer to a customer question.
Question: {q}
Reference facts: {ref}
Answer: {a}
Check each criterion; reason briefly first, then output JSON.
1. All claims are supported by the reference facts (no fabrication).
2. Every reference fact needed to answer is included.
3. Tone is professional.
Return: {{"reasoning": "...", "supported": true|false, "complete": true|false, "tone_ok": true|false}}"""
  1. Write a specific rubric with binary or few-level criteria; ask for reasoning before the verdict.
  2. Calibrate against humans: label 100+ items yourself, compute agreement (accuracy, Cohen’s κ\kappa) with the judge, iterate on the rubric until it’s high enough for your purpose. Re-check when you change the judge model.
  3. Randomize order, use temperature 0 or multiple samples, and track judge variance.
  4. Use the judge for trends and regressions, human review for ship decisions on high-stakes tasks.
  5. Fine-tuned small judges and reward models can be cheaper at scale; validate the same way.
  6. Beware circularity: if you optimize your system against the same judge (or use it as an RL reward), it will learn to please it (Post-Training).

Needed for ground truth, calibrating automatic metrics, and subjective quality.

  • Define criteria concretely (accuracy, completeness, tone, harm) with guidelines and examples, not “is it good?”.
  • Pairwise comparison with randomized order and optional ties is more reliable than absolute Likert scores.
  • Use multiple raters per item (3 or more), measure inter-annotator agreement (Cohen’s or Fleiss’ κ\kappa, Krippendorff’s α\alpha). Low agreement means the task or guidelines are ambiguous, not that humans are broken; fix the rubric before collecting more.
  • Expertise: domain tasks (medical, legal, code) need expert raters; crowd raters can’t verify and will favor fluent-sounding answers.
  • Blind raters to which system produced each output; mix in gold/attention-check items.
  • Sample size: with binary outcomes, the 95% interval half-width is about 1.96p(1−p)/n1.96\sqrt{p(1-p)/n}: at p=0.7p=0.7, n=100n=100 gives ±9\pm9 points; n=400n=400 gives ±4.5\pm4.5. For paired comparisons use paired tests (sign test, McNemar) or bootstrap over items.
  • Online signals: thumbs, edits, regeneration, task completion, escalations, A/B tests. Noisy but real; combine with offline evals.

Task-specific evals and eval-driven development

Section titled “Task-specific evals and eval-driven development”

Build your own eval set. The highest-leverage investment in an LLM project.

  1. Collect real inputs: production logs (with privacy review), support tickets, user interviews. Cover the distribution and the tail: typos, adversarial, ambiguous, multilingual, long, empty, out-of-scope.
  2. Define success per case: expected output, or a rubric, or a programmatic checker. Write down what a failure looks like.
  3. Start small: 50-200 well-chosen cases beat 10,000 sloppy ones. Grow it from every production failure (“every bug becomes a test case”).
  4. Include negatives: cases where the right behavior is to refuse, abstain or ask a clarifying question.
  5. Split: a development set you look at and tune against, and a held-out set you rarely touch to detect overfitting your prompts to the dev set.
  6. Automate: a single command runs all cases, computes metrics, and diffs against the previous run; wire it into CI for prompt, model, retrieval and code changes.
  7. Slice results by category, difficulty, language, user segment, to catch regressions hidden in averages.
  8. Version everything: prompts, model IDs, parameters, dataset versions, judge prompts.
import json, statistics as st
from concurrent.futures import ThreadPoolExecutor
def run_eval(system, cases, check, n_samples=3, workers=8):
"""system(input)->output; check(case, output)->bool. Returns pass rate and per-case results."""
def one(case):
passes = [check(case, system(case["input"])) for _ in range(n_samples)]
return {"id": case["id"], "tags": case.get("tags", []), "pass_rate": sum(passes) / n_samples}
with ThreadPoolExecutor(workers) as ex:
results = list(ex.map(one, cases))
overall = st.mean(r["pass_rate"] for r in results)
return overall, results
# compare two versions with the same cases; inspect the per-case flips, not just the mean

Eval-driven development is test-driven development for LLM systems: write evals for the behavior you want before tuning prompts; change one thing at a time; look at the failures (read transcripts: error analysis is where the insight is); categorize failure modes; fix the largest category; re-run. Maintain a taxonomy of failures so effort goes where the metric moves.

Nondeterminism. Run each case multiple times (or use temperature 0, acknowledging residual nondeterminism), report pass rate and variance, and for agents distinguish pass@1 from reliability (all of kk runs succeed). Use paired comparisons between versions, with bootstrap confidence intervals; don’t celebrate a 2-point gain on 100 cases.

Pipeline decomposition. Evaluate components separately (retrieval, reranker, generator, tool selection, judge) and end-to-end, so you can localize regressions (RAG, Agents).

Production monitoring. Offline evals miss distribution shift. Sample live traffic for review, run lightweight judges or heuristics online (refusal rate, format failure, latency, cost, user feedback), alert on drift, and feed failures back into the eval set (Building LLM Applications).

Red-teaming is adversarial evaluation: deliberately trying to make the system fail in harmful, insecure or embarrassing ways, to find problems before attackers or users do.

Areas:

  • Harmful content and misuse: weapons, cyber, self-harm, harassment, illegal advice, regulated domains.
  • Jailbreaks: role-play framings, hypothetical wrappers, encoding/obfuscation (base64, other languages, leetspeak), many-shot jailbreaking (flooding long context with examples), gradual multi-turn escalation, adversarial suffixes found by optimization.
  • Prompt injection and data exfiltration against RAG and agents (Prompting, Agents): injected instructions in documents, tool outputs, web pages, images.
  • Privacy: training-data extraction, system-prompt leakage, PII disclosure, cross-tenant leakage.
  • Bias and fairness: disparate quality or tone across groups; stereotype reinforcement.
  • Hallucination and overconfidence, sycophancy, deceptive behavior in agentic settings, excessive agency (taking unrequested actions).
  • Over-refusal: refusing benign requests is a failure too; test borderline-benign prompts.

Methods:

  1. Manual expert red-teaming: creative humans with domain knowledge find novel failures. Scope, rules of engagement and a vulnerability taxonomy keep it productive.
  2. Automated red-teaming: use an attacker LLM to generate and iteratively refine attacks against the target, with a classifier or judge scoring success; plus optimization-based attacks for open-weight models. Scales coverage and enables regression suites.
  3. Benchmarked suites (HarmBench-style, jailbreak benchmarks) for tracking attack success rate (ASR) over time, with the caveat that benchmarks get trained against.
  4. System-level testing: attack the whole application (tools, permissions, filters, retrieval), not just the model. Many real vulnerabilities live in glue code.
  5. Continuous: re-run on every model, prompt or tool change; add every discovered exploit to a regression set; run bug-bounty style programs.

Report attack success rate with uncertainty by category and severity, and track mitigations: model training (Post-Training), input/output classifiers, permissions, monitoring. A clean red-team result shows only that those attacks failed; adversaries adapt.

  • Public benchmarks are a coarse, contaminable, harness-sensitive filter; your own task-specific eval set is what predicts success.
  • Prefer verifiable metrics (tests, schema checks, exact match); use LLM judges for the rest, with rubrics, order randomization, and calibration against human labels.
  • Human evaluation needs clear criteria, multiple raters, agreement metrics and enough samples; mind sample-size arithmetic.
  • Eval-driven development: build evals from real data, grow them from failures, slice results, version everything, run in CI, monitor in production.
  • Handle nondeterminism with repeated runs and paired statistics.
  • Red-team the whole system continuously: jailbreaks, injection, leakage, over-refusal; track attack success rate and regress-test fixes.
  1. Hands-on. Create 60 labelled cases for a task you care about. Evaluate two prompt variants with 3 samples each and report paired bootstrap 95% confidence intervals for the difference.
  2. Conceptual. A benchmark score jumps 8 points between two model versions. List three non-capability explanations to rule out.
  3. Hands-on. Build a pairwise LLM judge, then measure its position bias by swapping answer order on 100 pairs. What fraction of verdicts flip?
  4. Conceptual. Compute the 95% interval for an accuracy of 82% on 150 items. Is a model at 86% on the same items significantly better?
  5. Hands-on. Write 20 prompt-injection test cases for a RAG app (documents containing instructions) and an automated checker for compliance; report the attack success rate before and after adding a mitigation.
Solutions (sketch)
  1. Contamination or post-training on benchmark-like data; a changed prompt/extraction harness or sampling setting; different reasoning budget/tool access; label noise fixes; evaluation on a different split; lucky variance on a small set.
  2. 1.960.82⋅0.18/150≈0.0611.96\sqrt{0.82\cdot0.18/150}\approx0.061, so 82% ±\pm 6.1 points. The intervals overlap; since the items are shared, a paired test is more powerful, but a 4-point gap on 150 items is generally not significant unless the models’ errors are strongly correlated and the paired test supports it.
  • Zheng et al., “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena” (2023).
  • Liang et al., “Holistic Evaluation of Language Models” (HELM, 2022).
  • Hendrycks et al., “Measuring Massive Multitask Language Understanding” (2020); Chen et al., “Evaluating Large Language Models Trained on Code” (HumanEval, 2021); Jimenez et al., “SWE-bench” (2023).
  • Jacovi et al., “Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarks” (2023).
  • Perez et al., “Red Teaming Language Models with Language Models” (2022); Ganguli et al., “Red Teaming Language Models to Reduce Harms” (2022); Mazeika et al., “HarmBench” (2024).
  • Zou et al., “Universal and Transferable Adversarial Attacks on Aligned Language Models” (2023).
  • Hamel Husain’s writing on LLM evals; Chip Huyen, AI Engineering (2025), evaluation chapters.
  • Miller, “Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations” (2024).