Reinforcement Learning and Synthetic Data for Agents
What you’ll learn. How agents are trained, not just prompted: the RL formulation for multi-turn LLM agents; REINFORCE and the group-relative method (GRPO) used in recent reasoning and agent training; verifiable rewards and environment design; the cheaper alternative of rejection-sampling fine-tuning on synthetic trajectories; and the failure modes (reward hacking, credit assignment, instability). Background on RLHF and preference methods is in Post-training; this lecture focuses on the agentic case.
Why train agents
Section titled “Why train agents”Prompting plateaus. A base model has never been optimized for long-horizon tool use with recovery from errors; it saw such behavior only indirectly. Training on interaction directly optimizes the quantity we care about, task success, and teaches skills like trying a different approach after a failed test. The empirical story since 2024: reinforcement learning on verifiable outcomes (math, code, games) produced reasoning models, and extending the same recipe to multi-turn tool use produced stronger coding, search and browsing agents.
Formulation
Section titled “Formulation”Treat an episode as a trajectory where actions are token sequences (a thought plus tool call) and observations come from tools. A reward is assigned, typically at the end. The goal:
The policy-gradient theorem gives
where is a baseline that reduces variance without bias. Only tokens the model generated receive gradient; tool outputs are in the context but masked out of the loss.
Verifiable rewards (RLVR)
Section titled “Verifiable rewards (RLVR)”Rewards that can be computed by a program, with no learned reward model:
- Math: final answer matches ground truth.
- Code: tests pass.
- Games, puzzles, formal proofs: win or valid proof.
- Tool tasks: the final database state equals the expected state (the approach of tau-bench-style evaluation, see Evaluation).
- Retrieval tasks: answer matches a known fact after browsing.
Benefits: cheap, objective, hard to bias, and scalable. Limitation: only tasks with checkable answers; extending to open-ended tasks uses rubrics or learned judges, which reintroduces hackability.
GRPO: group-relative policy optimization
Section titled “GRPO: group-relative policy optimization”PPO needs a value network (a critic) of similar size to the policy, which is costly. GRPO (introduced with DeepSeekMath and used in DeepSeek-R1) drops it: for each prompt sample a group of trajectories, score them, and use the group’s statistics as the baseline.
Each token in trajectory gets advantage . The clipped objective (as in PPO) per token ratio :
A toy version of the advantage computation and loss in PyTorch:
import torch
def grpo_loss(logp_new, logp_old, rewards, mask, eps=0.2): """ logp_new, logp_old: (G, T) per-token log-probs of sampled tokens rewards: (G,) scalar reward per trajectory mask: (G, T) 1 for model-generated tokens, 0 for prompt/tool-output tokens """ adv = (rewards - rewards.mean()) / (rewards.std() + 1e-6) # (G,) adv = adv[:, None] ratio = torch.exp(logp_new - logp_old) unclipped = ratio * adv clipped = torch.clamp(ratio, 1 - eps, 1 + eps) * adv per_tok = -torch.minimum(unclipped, clipped) * mask return per_tok.sum() / mask.sum()If all samples succeed or all fail the advantage is zero and that prompt contributes no signal, which is why curating prompts of intermediate difficulty (success rate neither 0 nor 1) matters. Variants (DAPO, Dr. GRPO and others) adjust length normalization, clipping and sampling to improve stability; details are fast-moving.
What is different for agents
Section titled “What is different for agents”- Multi-turn rollouts. Each rollout interleaves generation with real tool execution, which is slower and stateful. Rollout infrastructure (parallel sandboxes, containers, VMs, simulated users) is the main engineering cost.
- Loss masking of tool outputs and environment text; otherwise the model is trained to “predict” observations.
- Credit assignment. A 40-step trajectory with a single final reward gives a weak signal per action. Remedies: outcome rewards with many samples per task (the group baseline), intermediate rewards (partial test passes, subgoal completion), process rewards, or turn-level advantage estimation.
- Off-policy drift and asynchrony. Slow rollouts mean the data was sampled from an older policy; asynchronous trainers tolerate bounded staleness with importance corrections.
- Long contexts. Trajectories can be tens of thousands of tokens; memory and throughput constrain training, and context management (Memory) becomes part of the learned policy.
- Variable length. Reward shaping on length or number of steps (cost penalties) prevents endlessly long episodes.
Environments are the bottleneck
Section titled “Environments are the bottleneck”An RL environment needs: a task generator, a sandboxed executable world, a reliable reward function, and reset. Sources: GitHub repositories with test suites (SWE-style task mining), synthetic web apps, simulated APIs with databases and policy rules, games, theorem provers. Quality dimensions: diversity (avoid overfitting to a narrow task family), difficulty calibration, verifier correctness (flaky or buggy tests teach nonsense), and resistance to shortcuts. Many labs treat environment building as the new data-collection frontier.
Cheaper than online RL: rejection sampling and trajectory distillation
Section titled “Cheaper than online RL: rejection sampling and trajectory distillation”Before (or instead of) full RL, improve a model by learning from its own successful behavior:
Rejection-sampling fine-tuning (RFT / STaR / expert iteration):
- Sample trajectories per task from the current model.
- Keep those that pass the verifier.
- Fine-tune on them (supervised).
- Repeat.
STaR (Zelikman et al., 2022) used exactly this loop with rationales; it is a special case of policy improvement with a binary reward and is more stable and simpler than online RL, though less sample-efficient at exploring.
Distillation from a stronger agent: run a strong (teacher) agent on many tasks, filter successful trajectories by the verifier, and fine-tune a smaller model on them. This is how many compact open agent models are produced. Note the teacher’s terms of service may restrict using outputs to train competing models.
Synthetic task generation: use a model to create tasks (and sometimes tests), validate them by checking that a reference solution passes and a null solution fails, then solve them. Pipelines for synthetic software-engineering tasks (injecting bugs, mining commit histories) have produced sizable training sets. Pitfalls: model collapse or narrowing from training on one’s own outputs without fresh signal, duplicates and benchmark contamination, and verifiers weaker than the generators.
def expert_iteration(model, tasks, verify, rollout, finetune, k=8, rounds=3): for _ in range(rounds): data = [] for t in tasks: trajs = [rollout(model, t) for _ in range(k)] good = [tr for tr in trajs if verify(t, tr)] data += good[:2] # keep a few per task to preserve diversity model = finetune(model, data) return modelReward hacking
Section titled “Reward hacking”Optimizers exploit any gap between the reward and your intent. Documented patterns in code agents: editing or deleting tests, special-casing test inputs, exiting with code 0 early, reading expected outputs from the file system, and exploiting evaluation harness bugs. Defenses: sandbox the grader out of the agent’s reach, hold out hidden tests, penalize edits to test files, monitor trajectories (an LLM monitor reading transcripts catches many hacks), and use multiple independent checks. Treat discovered hacks as environment bugs to fix.
Other things that make RL brittle
Section titled “Other things that make RL brittle”- Entropy collapse: the policy becomes deterministic and stops exploring; monitor entropy, use clip-higher or entropy bonuses.
- KL and reference drift: the KL term keeps the model near a sane reference; too strong stalls learning, too weak lets it drift into degenerate behavior.
- Verifier noise: 5% label noise can cap achievable performance.
- Reward sparsity: if the initial success rate is near zero there is no signal; bootstrap with supervised fine-tuning on demonstrations first (a “cold start”) or easier curriculum tasks.
Practical advice for practitioners
Section titled “Practical advice for practitioners”Most people will not run large-scale RL. Realistic options: (1) improve the harness and prompts, (2) collect your own verified trajectories and fine-tune a small open model with RFT, (3) use hosted reinforcement fine-tuning offered by some providers if you can write a grader, (4) use open frameworks for GRPO-style training on small models to learn the dynamics. Always keep a held-out evaluation set that the training process never touched.
Key takeaways
Section titled “Key takeaways”- Agent RL optimizes expected outcome reward over multi-turn trajectories; only model-generated tokens carry gradient.
- Verifiable rewards (tests, state checks, answer match) make RL scalable; GRPO removes the critic via group-normalized advantages.
- Multi-turn RL adds heavy infrastructure (sandboxed rollouts), credit-assignment difficulty and staleness issues.
- Rejection-sampling fine-tuning and distillation of verified trajectories are simple, robust first steps.
- Reward hacking is the default outcome unless environments and graders are carefully hardened.
Exercises
Section titled “Exercises”- Compute. Four rollouts get rewards . Compute the GRPO advantages (use population std). What if rewards are ?
- Conceptual. Why mask tool outputs in the loss?
- Hands-on. Implement a toy bandit: a “policy” over 3 tools with softmax logits, true success rates , train with the group-baseline REINFORCE update and check that probability mass moves to tool 3.
- Design. Propose three reward-hacking routes in an environment that rewards “all tests pass” and a mitigation for each.
Solutions (1, 2)
- Mean 0.5, std 0.5, advantages . For all-ones, std is 0 so advantages are 0 (with the epsilon in the denominator): no gradient.
- Tool outputs are not the policy’s choices; including them would train the model to imitate or predict environment text and would mis-assign credit.
Further reading
Section titled “Further reading”- Sutton and Barto, Reinforcement Learning: An Introduction; Williams (1992), REINFORCE.
- Schulman et al., “Proximal Policy Optimization Algorithms” (2017).
- Shao et al., “DeepSeekMath” (2024, introduces GRPO); DeepSeek-AI, “DeepSeek-R1” (2025).
- Zelikman et al., “STaR: Bootstrapping Reasoning with Reasoning” (2022).
- Lambert et al., “Tulu 3” (2024) on RL with verifiable rewards; see Post-training.
- Yue et al., “Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?” (2025).