Skip to content

Reinforcement Learning and Synthetic Data for Agents

What you’ll learn. How agents are trained, not just prompted: the RL formulation for multi-turn LLM agents; REINFORCE and the group-relative method (GRPO) used in recent reasoning and agent training; verifiable rewards and environment design; the cheaper alternative of rejection-sampling fine-tuning on synthetic trajectories; and the failure modes (reward hacking, credit assignment, instability). Background on RLHF and preference methods is in Post-training; this lecture focuses on the agentic case.

Prompting plateaus. A base model has never been optimized for long-horizon tool use with recovery from errors; it saw such behavior only indirectly. Training on interaction directly optimizes the quantity we care about, task success, and teaches skills like trying a different approach after a failed test. The empirical story since 2024: reinforcement learning on verifiable outcomes (math, code, games) produced reasoning models, and extending the same recipe to multi-turn tool use produced stronger coding, search and browsing agents.

Treat an episode as a trajectory τ=(s0,a0,o1,a1,…,aT)\tau = (s_0, a_0, o_1, a_1, \dots, a_T) where actions ata_t are token sequences (a thought plus tool call) and observations oto_t come from tools. A reward R(τ)R(\tau) is assigned, typically at the end. The goal:

J(θ)=Eτ∼πθ[R(τ)].J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta}\left[ R(\tau) \right].

The policy-gradient theorem gives

∇θJ=Eτ[∑t∇θlog⁡πθ(at∣ht) (R(τ)−b)],\nabla_\theta J = \mathbb{E}_\tau\left[ \sum_t \nabla_\theta \log \pi_\theta(a_t \mid h_t)\,\big(R(\tau) - b\big)\right],

where bb is a baseline that reduces variance without bias. Only tokens the model generated receive gradient; tool outputs are in the context but masked out of the loss.

Rewards that can be computed by a program, with no learned reward model:

  • Math: final answer matches ground truth.
  • Code: tests pass.
  • Games, puzzles, formal proofs: win or valid proof.
  • Tool tasks: the final database state equals the expected state (the approach of tau-bench-style evaluation, see Evaluation).
  • Retrieval tasks: answer matches a known fact after browsing.

Benefits: cheap, objective, hard to bias, and scalable. Limitation: only tasks with checkable answers; extending to open-ended tasks uses rubrics or learned judges, which reintroduces hackability.

PPO needs a value network (a critic) of similar size to the policy, which is costly. GRPO (introduced with DeepSeekMath and used in DeepSeek-R1) drops it: for each prompt sample a group of GG trajectories, score them, and use the group’s statistics as the baseline.

Ai=ri−mean⁡(r1,…,rG)std⁡(r1,…,rG)A_i = \frac{r_i - \operatorname{mean}(r_1,\dots,r_G)}{\operatorname{std}(r_1,\dots,r_G)}

Each token in trajectory ii gets advantage AiA_i. The clipped objective (as in PPO) per token ratio ρ=πθ/πθold\rho = \pi_\theta / \pi_{\theta_\text{old}}:

L=−E[min⁡(ρA, clip⁡(ρ,1−ϵ,1+ϵ)A)]+β KL(πθ ∥ πref).\mathcal{L} = -\mathbb{E}\left[\min\big(\rho A,\ \operatorname{clip}(\rho, 1-\epsilon, 1+\epsilon) A\big)\right] + \beta\, \mathrm{KL}(\pi_\theta \,\|\, \pi_\text{ref}).

A toy version of the advantage computation and loss in PyTorch:

import torch
def grpo_loss(logp_new, logp_old, rewards, mask, eps=0.2):
"""
logp_new, logp_old: (G, T) per-token log-probs of sampled tokens
rewards: (G,) scalar reward per trajectory
mask: (G, T) 1 for model-generated tokens, 0 for prompt/tool-output tokens
"""
adv = (rewards - rewards.mean()) / (rewards.std() + 1e-6) # (G,)
adv = adv[:, None]
ratio = torch.exp(logp_new - logp_old)
unclipped = ratio * adv
clipped = torch.clamp(ratio, 1 - eps, 1 + eps) * adv
per_tok = -torch.minimum(unclipped, clipped) * mask
return per_tok.sum() / mask.sum()

If all GG samples succeed or all fail the advantage is zero and that prompt contributes no signal, which is why curating prompts of intermediate difficulty (success rate neither 0 nor 1) matters. Variants (DAPO, Dr. GRPO and others) adjust length normalization, clipping and sampling to improve stability; details are fast-moving.

  1. Multi-turn rollouts. Each rollout interleaves generation with real tool execution, which is slower and stateful. Rollout infrastructure (parallel sandboxes, containers, VMs, simulated users) is the main engineering cost.
  2. Loss masking of tool outputs and environment text; otherwise the model is trained to “predict” observations.
  3. Credit assignment. A 40-step trajectory with a single final reward gives a weak signal per action. Remedies: outcome rewards with many samples per task (the group baseline), intermediate rewards (partial test passes, subgoal completion), process rewards, or turn-level advantage estimation.
  4. Off-policy drift and asynchrony. Slow rollouts mean the data was sampled from an older policy; asynchronous trainers tolerate bounded staleness with importance corrections.
  5. Long contexts. Trajectories can be tens of thousands of tokens; memory and throughput constrain training, and context management (Memory) becomes part of the learned policy.
  6. Variable length. Reward shaping on length or number of steps (cost penalties) prevents endlessly long episodes.

An RL environment needs: a task generator, a sandboxed executable world, a reliable reward function, and reset. Sources: GitHub repositories with test suites (SWE-style task mining), synthetic web apps, simulated APIs with databases and policy rules, games, theorem provers. Quality dimensions: diversity (avoid overfitting to a narrow task family), difficulty calibration, verifier correctness (flaky or buggy tests teach nonsense), and resistance to shortcuts. Many labs treat environment building as the new data-collection frontier.

Cheaper than online RL: rejection sampling and trajectory distillation

Section titled “Cheaper than online RL: rejection sampling and trajectory distillation”

Before (or instead of) full RL, improve a model by learning from its own successful behavior:

Rejection-sampling fine-tuning (RFT / STaR / expert iteration):

  1. Sample kk trajectories per task from the current model.
  2. Keep those that pass the verifier.
  3. Fine-tune on them (supervised).
  4. Repeat.

STaR (Zelikman et al., 2022) used exactly this loop with rationales; it is a special case of policy improvement with a binary reward and is more stable and simpler than online RL, though less sample-efficient at exploring.

Distillation from a stronger agent: run a strong (teacher) agent on many tasks, filter successful trajectories by the verifier, and fine-tune a smaller model on them. This is how many compact open agent models are produced. Note the teacher’s terms of service may restrict using outputs to train competing models.

Synthetic task generation: use a model to create tasks (and sometimes tests), validate them by checking that a reference solution passes and a null solution fails, then solve them. Pipelines for synthetic software-engineering tasks (injecting bugs, mining commit histories) have produced sizable training sets. Pitfalls: model collapse or narrowing from training on one’s own outputs without fresh signal, duplicates and benchmark contamination, and verifiers weaker than the generators.

def expert_iteration(model, tasks, verify, rollout, finetune, k=8, rounds=3):
for _ in range(rounds):
data = []
for t in tasks:
trajs = [rollout(model, t) for _ in range(k)]
good = [tr for tr in trajs if verify(t, tr)]
data += good[:2] # keep a few per task to preserve diversity
model = finetune(model, data)
return model

Optimizers exploit any gap between the reward and your intent. Documented patterns in code agents: editing or deleting tests, special-casing test inputs, exiting with code 0 early, reading expected outputs from the file system, and exploiting evaluation harness bugs. Defenses: sandbox the grader out of the agent’s reach, hold out hidden tests, penalize edits to test files, monitor trajectories (an LLM monitor reading transcripts catches many hacks), and use multiple independent checks. Treat discovered hacks as environment bugs to fix.

  • Entropy collapse: the policy becomes deterministic and stops exploring; monitor entropy, use clip-higher or entropy bonuses.
  • KL and reference drift: the KL term keeps the model near a sane reference; too strong stalls learning, too weak lets it drift into degenerate behavior.
  • Verifier noise: 5% label noise can cap achievable performance.
  • Reward sparsity: if the initial success rate is near zero there is no signal; bootstrap with supervised fine-tuning on demonstrations first (a “cold start”) or easier curriculum tasks.

Most people will not run large-scale RL. Realistic options: (1) improve the harness and prompts, (2) collect your own verified trajectories and fine-tune a small open model with RFT, (3) use hosted reinforcement fine-tuning offered by some providers if you can write a grader, (4) use open frameworks for GRPO-style training on small models to learn the dynamics. Always keep a held-out evaluation set that the training process never touched.

  • Agent RL optimizes expected outcome reward over multi-turn trajectories; only model-generated tokens carry gradient.
  • Verifiable rewards (tests, state checks, answer match) make RL scalable; GRPO removes the critic via group-normalized advantages.
  • Multi-turn RL adds heavy infrastructure (sandboxed rollouts), credit-assignment difficulty and staleness issues.
  • Rejection-sampling fine-tuning and distillation of verified trajectories are simple, robust first steps.
  • Reward hacking is the default outcome unless environments and graders are carefully hardened.
  1. Compute. Four rollouts get rewards (1,0,0,1)(1,0,0,1). Compute the GRPO advantages (use population std). What if rewards are (1,1,1,1)(1,1,1,1)?
  2. Conceptual. Why mask tool outputs in the loss?
  3. Hands-on. Implement a toy bandit: a “policy” over 3 tools with softmax logits, true success rates (0.2,0.5,0.8)(0.2, 0.5, 0.8), train with the group-baseline REINFORCE update and check that probability mass moves to tool 3.
  4. Design. Propose three reward-hacking routes in an environment that rewards “all tests pass” and a mitigation for each.
Solutions (1, 2)
  1. Mean 0.5, std 0.5, advantages (1,−1,−1,1)(1,-1,-1,1). For all-ones, std is 0 so advantages are 0 (with the epsilon in the denominator): no gradient.
  2. Tool outputs are not the policy’s choices; including them would train the model to imitate or predict environment text and would mis-assign credit.
  • Sutton and Barto, Reinforcement Learning: An Introduction; Williams (1992), REINFORCE.
  • Schulman et al., “Proximal Policy Optimization Algorithms” (2017).
  • Shao et al., “DeepSeekMath” (2024, introduces GRPO); DeepSeek-AI, “DeepSeek-R1” (2025).
  • Zelikman et al., “STaR: Bootstrapping Reasoning with Reasoning” (2022).
  • Lambert et al., “Tulu 3” (2024) on RL with verifiable rewards; see Post-training.
  • Yue et al., “Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?” (2025).