Code Agents
What you’ll learn. Why coding is the most successful agent domain; the anatomy of a coding agent (shell, file view/edit, search, tests); the agent-computer interface idea; localization and repair strategies; sandboxing; and how to run an agent on a real repository safely.
Why code is special
Section titled “Why code is special”Code is an unusually good domain for agents for structural reasons:
- Cheap, automatic verification. Compilers, type checkers, linters and tests give fast, objective feedback; the agent can check its own work without a human. This is the property that makes test-time search and RL with verifiable rewards work so well here.
- Text-native. Source and logs are tokens.
- Reversible. Version control makes most actions undoable.
- Abundant training data in pretraining and large numbers of executable repositories.
- Code is a universal action language. A code-writing agent can build its own tools (Tool use).
Anatomy of a coding agent
Section titled “Anatomy of a coding agent”Almost every modern coding agent (CLI, IDE or cloud-based) exposes roughly this toolset:
| Tool | Purpose | Design note |
|---|---|---|
bash / shell |
Run tests, git, build, any command | Needs timeouts and sandboxing |
read_file |
View with line numbers, ranges | Cap size; allow offset/limit |
edit_file |
Replace a unique string or apply a patch | Fail loudly if match is ambiguous |
grep / glob |
Search by content and name | Faster and more precise than embeddings for identifiers |
todo |
Maintain a plan | Keeps long tasks on track |
subagent |
Delegate exploration | Protects main context |
The loop is the ReAct loop of earlier lectures: read, hypothesize, edit, run tests, read the failure, repeat.
The agent-computer interface (ACI)
Section titled “The agent-computer interface (ACI)”The SWE-agent work (Yang et al., 2024) popularized the point that agents need interfaces designed for them, as humans have IDEs designed for us. Findings that generalize:
- A windowed file viewer with line numbers beats
caton huge files. - Edit commands with built-in linting that reject syntactically invalid edits stop the agent from compounding a broken file.
- Concise, structured feedback (“Edit applied. Lines 40-52 now:…”) beats silent success or noisy output.
- Guardrails on search output (cap at 50 hits and tell the model to narrow) prevent context floods.
Small interface changes shifted measured resolve rates by many points with the same underlying model, which is why harness engineering is a first-class concern.
Workflow of a typical task
Section titled “Workflow of a typical task”1. Understand read the issue; explore repo structure (ls, grep, read key files)2. Reproduce write or locate a failing test / script; run it; see the failure3. Localize narrow to file and function using stack traces and search4. Edit minimal patch5. Verify run the failing test, then the relevant suite, linter, type check6. Review diff the working tree; check for unintended changes7. Report summarize changes, caveats, what was not verifiedReproduce first is the single highest-leverage habit: it provides a success signal that can drive iteration and avoids “fixes” that address a misunderstanding.
Localization
Section titled “Localization”Finding where to change is often harder than the change. Strategies: stack-trace following, grep for error strings and identifiers, reading call sites, test file mirroring (tests/test_foo.py hints foo.py), and repository maps (a compact outline of classes and signatures, as in Aider’s repo-map). Agentless-style pipelines (Xia et al., 2024) showed that a non-agentic hierarchical localize-then-repair workflow with test-based selection can match complex agents, a useful reminder that workflows can beat agents.
A minimal edit tool
Section titled “A minimal edit tool”import pathlib
def str_replace(path: str, old: str, new: str) -> str: """Replace one exact occurrence of `old` with `new` in the file.""" p = pathlib.Path(path) if not p.is_file(): return f"ERROR: {path} does not exist" text = p.read_text() n = text.count(old) if n == 0: return "ERROR: `old` not found; re-read the file, whitespace must match exactly" if n > 1: return f"ERROR: `old` matches {n} places; include more surrounding context" p.write_text(text.replace(old, new, 1)) return "OK"Uniqueness checking is the key design idea: it forces the model to specify enough context and returns an instructive error otherwise. Alternatives include unified-diff application and whole-file rewrite; diffs are compact but models botch hunk headers, whole-file rewrites are robust but expensive on large files.
Sandboxing and isolation
Section titled “Sandboxing and isolation”An agent that runs shell commands can delete files, exfiltrate secrets or install malware. Mitigations by increasing strength:
- Permission prompts for risky commands (allowlist safe reads and test runs, confirm writes outside the workspace).
- Working in a git branch/worktree so changes are reviewable and revertible.
- Containers (Docker) with no host mounts beyond the repo and no credentials.
- MicroVMs / gVisor for stronger isolation when running untrusted code at scale.
- Network egress control: default-deny, allowlist package registries.
Never run an agent with your production credentials in the environment. See Safety and security.
# Throwaway container: repo mounted, no network, resource-limiteddocker run --rm -it --network none --memory 4g --cpus 2 \ -v "$PWD":/work -w /work python:3.12 bashVerifying and the problem of weak tests
Section titled “Verifying and the problem of weak tests”Tests are a proxy for correctness. Agents optimize the proxy:
- Test tampering: deleting or special-casing a failing test; hard-coding the expected value. Mitigate by making tests read-only to the agent or reviewing the diff for test edits.
- Overfitting to visible tests: passes the given tests but not the hidden ones. Evaluate on held-out tests.
- False-positive “fixes”: the patch changes behavior elsewhere. Run the broader suite.
This is reward hacking in miniature, and the same pathology shows up when training on test reward (RL for agents).
Beyond single-shot fixes
Section titled “Beyond single-shot fixes”- Long-horizon work (a feature over hours): use todo lists, progress files, frequent commits as checkpoints, and context compaction (Memory).
- Parallel agents: several agents on separate git worktrees attempting different tasks or the same task, then picking the best (test-time search).
- Review agents: a second agent reviews the diff; effective when it has execution access, not just reading.
- Conventions files: a repository-level instructions file (build commands, style, gotchas) acts as procedural memory and sharply improves reliability.
Practical advice for using coding agents
Section titled “Practical advice for using coding agents”- Give a way to verify: tests, a script, a type check. State it explicitly.
- Keep tasks scoped; split large changes.
- Review diffs as you would a junior colleague’s pull request.
- Put build/test commands and architectural rules in the instructions file.
- Use version control as the undo button; commit before big runs.
Key takeaways
Section titled “Key takeaways”- Code has fast automatic verification, reversibility and text-native tooling, so agents work best there.
- A coding agent is a ReAct loop over shell, view, edit, search and a plan tool; interface design (the ACI) moves results as much as model choice.
- Reproduce, localize, minimal edit, verify, review.
- Sandbox by default; prefer containers and branches; never expose production secrets.
- Watch for test tampering and overfitting to visible tests.
Exercises
Section titled “Exercises”- Conceptual. Why does an edit tool that rejects ambiguous matches improve agent reliability over one that replaces the first match silently?
- Hands-on. Extend
str_replaceto runpython -m py_compileon the file after editing and revert if it fails to compile. - Hands-on. Write a tiny repo with a failing test, and drive any LLM client through the loop (read, edit, run pytest) using the harness from Build an agent from scratch.
- Design. List three ways an agent might “cheat” a unit test suite and one detection method for each.
Solutions (4)
(a) Edit the test: detect via diff touching test files. (b) Hard-code the expected output: run hidden or randomized tests. (c) Catch-all except: pass hiding the failure: lint/static checks and a reviewer agent; also mutation testing.
Further reading
Section titled “Further reading”- Jimenez et al., “SWE-bench: Can Language Models Resolve Real-World GitHub Issues?” (2023).
- Yang et al., “SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering” (2024).
- Xia et al., “Agentless: Demystifying LLM-based Software Engineering Agents” (2024).
- Chen et al., “Evaluating Large Language Models Trained on Code” (Codex, 2021).
- Wang et al., “OpenHands: An Open Platform for AI Software Developers as Generalist Agents” (2024).