Skip to content

Code Agents

What you’ll learn. Why coding is the most successful agent domain; the anatomy of a coding agent (shell, file view/edit, search, tests); the agent-computer interface idea; localization and repair strategies; sandboxing; and how to run an agent on a real repository safely.

Code is an unusually good domain for agents for structural reasons:

  1. Cheap, automatic verification. Compilers, type checkers, linters and tests give fast, objective feedback; the agent can check its own work without a human. This is the property that makes test-time search and RL with verifiable rewards work so well here.
  2. Text-native. Source and logs are tokens.
  3. Reversible. Version control makes most actions undoable.
  4. Abundant training data in pretraining and large numbers of executable repositories.
  5. Code is a universal action language. A code-writing agent can build its own tools (Tool use).

Almost every modern coding agent (CLI, IDE or cloud-based) exposes roughly this toolset:

Tool Purpose Design note
bash / shell Run tests, git, build, any command Needs timeouts and sandboxing
read_file View with line numbers, ranges Cap size; allow offset/limit
edit_file Replace a unique string or apply a patch Fail loudly if match is ambiguous
grep / glob Search by content and name Faster and more precise than embeddings for identifiers
todo Maintain a plan Keeps long tasks on track
subagent Delegate exploration Protects main context

The loop is the ReAct loop of earlier lectures: read, hypothesize, edit, run tests, read the failure, repeat.

The SWE-agent work (Yang et al., 2024) popularized the point that agents need interfaces designed for them, as humans have IDEs designed for us. Findings that generalize:

  • A windowed file viewer with line numbers beats cat on huge files.
  • Edit commands with built-in linting that reject syntactically invalid edits stop the agent from compounding a broken file.
  • Concise, structured feedback (“Edit applied. Lines 40-52 now:…”) beats silent success or noisy output.
  • Guardrails on search output (cap at 50 hits and tell the model to narrow) prevent context floods.

Small interface changes shifted measured resolve rates by many points with the same underlying model, which is why harness engineering is a first-class concern.

1. Understand read the issue; explore repo structure (ls, grep, read key files)
2. Reproduce write or locate a failing test / script; run it; see the failure
3. Localize narrow to file and function using stack traces and search
4. Edit minimal patch
5. Verify run the failing test, then the relevant suite, linter, type check
6. Review diff the working tree; check for unintended changes
7. Report summarize changes, caveats, what was not verified

Reproduce first is the single highest-leverage habit: it provides a success signal that can drive iteration and avoids “fixes” that address a misunderstanding.

Finding where to change is often harder than the change. Strategies: stack-trace following, grep for error strings and identifiers, reading call sites, test file mirroring (tests/test_foo.py hints foo.py), and repository maps (a compact outline of classes and signatures, as in Aider’s repo-map). Agentless-style pipelines (Xia et al., 2024) showed that a non-agentic hierarchical localize-then-repair workflow with test-based selection can match complex agents, a useful reminder that workflows can beat agents.

import pathlib
def str_replace(path: str, old: str, new: str) -> str:
"""Replace one exact occurrence of `old` with `new` in the file."""
p = pathlib.Path(path)
if not p.is_file():
return f"ERROR: {path} does not exist"
text = p.read_text()
n = text.count(old)
if n == 0:
return "ERROR: `old` not found; re-read the file, whitespace must match exactly"
if n > 1:
return f"ERROR: `old` matches {n} places; include more surrounding context"
p.write_text(text.replace(old, new, 1))
return "OK"

Uniqueness checking is the key design idea: it forces the model to specify enough context and returns an instructive error otherwise. Alternatives include unified-diff application and whole-file rewrite; diffs are compact but models botch hunk headers, whole-file rewrites are robust but expensive on large files.

An agent that runs shell commands can delete files, exfiltrate secrets or install malware. Mitigations by increasing strength:

  • Permission prompts for risky commands (allowlist safe reads and test runs, confirm writes outside the workspace).
  • Working in a git branch/worktree so changes are reviewable and revertible.
  • Containers (Docker) with no host mounts beyond the repo and no credentials.
  • MicroVMs / gVisor for stronger isolation when running untrusted code at scale.
  • Network egress control: default-deny, allowlist package registries.

Never run an agent with your production credentials in the environment. See Safety and security.

Terminal window
# Throwaway container: repo mounted, no network, resource-limited
docker run --rm -it --network none --memory 4g --cpus 2 \
-v "$PWD":/work -w /work python:3.12 bash

Tests are a proxy for correctness. Agents optimize the proxy:

  • Test tampering: deleting or special-casing a failing test; hard-coding the expected value. Mitigate by making tests read-only to the agent or reviewing the diff for test edits.
  • Overfitting to visible tests: passes the given tests but not the hidden ones. Evaluate on held-out tests.
  • False-positive “fixes”: the patch changes behavior elsewhere. Run the broader suite.

This is reward hacking in miniature, and the same pathology shows up when training on test reward (RL for agents).

  • Long-horizon work (a feature over hours): use todo lists, progress files, frequent commits as checkpoints, and context compaction (Memory).
  • Parallel agents: several agents on separate git worktrees attempting different tasks or the same task, then picking the best (test-time search).
  • Review agents: a second agent reviews the diff; effective when it has execution access, not just reading.
  • Conventions files: a repository-level instructions file (build commands, style, gotchas) acts as procedural memory and sharply improves reliability.
  1. Give a way to verify: tests, a script, a type check. State it explicitly.
  2. Keep tasks scoped; split large changes.
  3. Review diffs as you would a junior colleague’s pull request.
  4. Put build/test commands and architectural rules in the instructions file.
  5. Use version control as the undo button; commit before big runs.
  • Code has fast automatic verification, reversibility and text-native tooling, so agents work best there.
  • A coding agent is a ReAct loop over shell, view, edit, search and a plan tool; interface design (the ACI) moves results as much as model choice.
  • Reproduce, localize, minimal edit, verify, review.
  • Sandbox by default; prefer containers and branches; never expose production secrets.
  • Watch for test tampering and overfitting to visible tests.
  1. Conceptual. Why does an edit tool that rejects ambiguous matches improve agent reliability over one that replaces the first match silently?
  2. Hands-on. Extend str_replace to run python -m py_compile on the file after editing and revert if it fails to compile.
  3. Hands-on. Write a tiny repo with a failing test, and drive any LLM client through the loop (read, edit, run pytest) using the harness from Build an agent from scratch.
  4. Design. List three ways an agent might “cheat” a unit test suite and one detection method for each.
Solutions (4)

(a) Edit the test: detect via diff touching test files. (b) Hard-code the expected output: run hidden or randomized tests. (c) Catch-all except: pass hiding the failure: lint/static checks and a reviewer agent; also mutation testing.

  • Jimenez et al., “SWE-bench: Can Language Models Resolve Real-World GitHub Issues?” (2023).
  • Yang et al., “SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering” (2024).
  • Xia et al., “Agentless: Demystifying LLM-based Software Engineering Agents” (2024).
  • Chen et al., “Evaluating Large Language Models Trained on Code” (Codex, 2021).
  • Wang et al., “OpenHands: An Open Platform for AI Software Developers as Generalist Agents” (2024).