Skip to content

Self-Refinement and Learning from Feedback

What you’ll learn. The family of self-improvement techniques that operate on a fixed model at inference time or via light training signals: Self-Refine, Reflexion, Constitutional AI, LLM-as-judge, verifiers and reward models, and learning from tool and execution feedback; when each works, and why intrinsic self-correction often does not.

“Self-improving agent” covers several loops at different timescales:

Loop What changes Timescale This course
Within an answer The output seconds this lecture
Within an episode The trajectory (retry, replan) minutes this lecture, patterns
Across episodes Memory, prompts, skills days memory
Across training runs Weights weeks test-time compute, RL

A unifying idea: improvement requires a signal that is better than the generator’s own first guess. Where does the signal come from? That question organizes this lecture and the next three.

Self-Refine (Madaan et al., 2023) iterates a single model through three roles: generate, feedback (critique with specific, actionable points), refine (rewrite given the feedback), until a stopping criterion.

def self_refine(llm, task, max_iters=3):
out = llm(task)
for _ in range(max_iters):
fb = llm(f"Task: {task}\nAnswer: {out}\nGive specific, actionable feedback. "
"If nothing important remains, say DONE.")
if "DONE" in fb[:20]:
break
out = llm(f"Task: {task}\nAnswer: {out}\nFeedback: {fb}\nRewrite the answer.")
return out

Reported gains were clear on subjective tasks (writing style, dialogue, code readability) and weaker on reasoning. Critique quality is the bottleneck.

When a model must correct without any external information, results are mixed to negative on reasoning. Huang et al. (2023, “Large Language Models Cannot Self-Correct Reasoning Yet”) showed that on math and QA, prompting a model to “review and fix” its answer, absent oracle knowledge of correctness, often leaves accuracy flat or lowers it: the model changes right answers to wrong ones about as often as the reverse. Gains previously reported were frequently due to leaking the ground truth into the stopping rule.

Principle: self-critique helps when verification is easier than generation and the verifier has information or compute the generator lacked (a test result, a retrieved fact, a reasoning trace, a different model). This is the generation-verification gap. It is large for code (run it), math with checkable answers, and constraint satisfaction; small for open-ended facts the model never knew.

Strong signals, roughly in order of reliability:

  1. Execution feedback: tests, compiler errors, runtime exceptions (Self-Debugging, Chen et al., 2023; code interpreters).
  2. Formal checkers: proof assistants, unit-conversion checkers, SQL execution, schema validation.
  3. Retrieved evidence: check claims against sources (CRITIC, Gou et al., 2023, uses tools like search to verify then revise).
  4. Environment reward: the task succeeded or not.
  5. A different or stronger model as critic or judge.
  6. Same-model critique: weakest; useful mostly for style and instruction-following checks.

Reflexion (Shinn et al., 2023) turns a failed attempt into a verbal lesson stored in episodic memory and prepended to the next try: “I failed because I searched with too generic a query; next time include the year.” No weights change; the “learning” is in context. It works when failure is detectable (tests failing, game lost) and trials can be repeated. It echoes human deliberate practice and anticipates modern “memory tools” in agents.

def reflexion(llm, run_attempt, task, tries=3):
lessons = []
for _ in range(tries):
result, ok, trace = run_attempt(task, lessons)
if ok:
return result
lessons.append(llm(f"Task: {task}\nTrace:\n{trace}\n"
"In two sentences, why did this fail and what to do differently?"))
return result

Constitutional AI (Bai et al., 2022) uses self-critique at training time. A model is given a written list of principles (the “constitution”); it critiques and revises its own responses against them, and the revised responses become supervised fine-tuning data (the SL-CAI phase). Then AI-generated preference labels (which of two responses better follows the principles) train a reward model for RL (RLAIF: reinforcement learning from AI feedback). The impact: scalable oversight where human labeling is replaced by model feedback guided by explicit rules, reducing reliance on human harm labeling. Details of preference optimization are in Post-training.

To use model feedback at scale you need scorers.

  • LLM-as-judge: prompt a strong model to grade outputs against a rubric or compare pairs. Cheap and flexible, but biased: position bias (prefers first answer), verbosity bias, self-preference (favors its own style). Mitigate with randomized order, rubrics, calibrated scales, and checking against human labels (Evaluation).
  • Reward models (RMs): trained on human or AI preference pairs to output a scalar. Outcome reward models score final answers; process reward models (PRMs) score each reasoning step (Lightman et al., 2023, “Let’s Verify Step by Step”) and can guide search, but need step-level labels.
  • Verifiers: models trained to judge correctness of a candidate solution (Cobbe et al., 2021 trained verifiers to rerank GSM8K samples). Generative verifiers produce a rationale then a verdict.
  • Programmatic verifiers: exact-match answers, unit tests, checkers; no learning, no bias, limited coverage.

Overoptimizing against a learned scorer is reward hacking (Goodhart’s law): the generator finds outputs that fool the scorer. The scorer’s quality limits how much optimization pressure is safe; use KL regularization in training and held-out evaluation.

Learning from tool and environment feedback

Section titled “Learning from tool and environment feedback”

Agents get feedback every step: a tool error, a failing test, a 404. Good agents treat observations as evidence and update the plan. Engineering tips:

  • Make tool errors informative (Tool use).
  • Add checkpoints: after each milestone, run a cheap check.
  • Detect loops (same action repeated 3 times) and force a strategy change or escalate to a human.
  • Persist lessons: write them to a notes file or memory the next session loads.

Self-improving beyond prompts: skills and code

Section titled “Self-improving beyond prompts: skills and code”

Agents can improve themselves as artifacts: accumulating a library of working code snippets or “skills” (Voyager, Wang et al., 2023, in Minecraft, stored verified programs in a skill library); rewriting their own prompts (automatic prompt optimization such as APE, OPRO, DSPy-style compilers); even editing their own scaffolding code (Gödel-agent and Darwin Gödel Machine-style research, 2024-2025, which evolve coding agents by empirically testing modifications on benchmarks). These are promising and fast-moving but narrow; they depend on a trustworthy evaluation signal, and every claim of “open-ended self-improvement” should be read with that in mind.

  • What is the signal? If you cannot name an external or asymmetric (verification easier than generation) signal, expect little gain.
  • Is there a stopping rule not based on the model’s own confidence?
  • Bound iterations; measure marginal gain per round (it usually collapses after 1-3).
  • Evaluate with and without refinement on held-out tasks; self-refinement can silently degrade.
  • Log critique and revision for audit.
  • Improvement needs a signal better than the generator’s first guess; the generation-verification gap decides success.
  • Intrinsic self-correction without external information often fails on reasoning; execution, tools, retrieval or a stronger judge fix this.
  • Reflexion stores lessons from failed attempts in memory; Constitutional AI uses self-critique to produce training data and AI preference labels.
  • LLM judges and reward models are biased and exploitable; verifiable programmatic checks are the gold standard.
  • Always evaluate refinement loops against a no-refinement baseline.
  1. Conceptual. Explain why Self-Refine helps on code with tests but little on factual trivia.
  2. Hands-on. Implement a refine loop for “write parse_date” with a test suite as feedback; plot pass rate per iteration over 20 tasks.
  3. Conceptual. A judge model prefers longer answers. Describe an experiment to measure this bias and one fix.
  4. Design. Design a Reflexion-style memory for a customer-support agent: what do you store, and how do you stop wrong lessons accumulating?
Solutions (3)

Take pairs of answers with equal correctness but different length (pad one with fluff); measure how often the judge picks the longer one beyond 50%. Fix: length-controlled rubric, instruct to ignore length, or normalize length statistically.

  • Madaan et al., “Self-Refine: Iterative Refinement with Self-Feedback” (2023).
  • Huang et al., “Large Language Models Cannot Self-Correct Reasoning Yet” (2023).
  • Shinn et al., “Reflexion: Language Agents with Verbal Reinforcement Learning” (2023).
  • Bai et al., “Constitutional AI: Harmlessness from AI Feedback” (2022).
  • Lightman et al., “Let’s Verify Step by Step” (2023); Cobbe et al., “Training Verifiers to Solve Math Word Problems” (2021).
  • Zheng et al., “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena” (2023).