Self-Refinement: Critique, Revision, and Stopping Policies

A model’s first draft is rarely its best. Self-refinement asks the model to critique its own output and revise it, repeating until a stopping condition is met. The central engineering question is not “does refinement help?” but “what kind of feedback, on what task, at what cost?” This lesson compares model self-feedback with external tool checks, shows cases where revision helps and where it harms, and defines a stopping policy.

The one-sentence version: Generate–critique–revise can improve open-ended and code-quality outputs using a model’s own feedback, but for verifiable factual or math tasks the model’s self-critique is unreliable and can make answers worse; external tool checks (search, code interpreter, API) are the more dependable correction signal, and every extra iteration costs tokens and latency.

1. The generate–critique–revise loop

One iteration has three steps:

1. Generate: produce an initial output y₀ from input x.
2. Critique: produce feedback fb on y₀.
3. Revise: produce a new output y₁ from x, y₀, and fb.

The loop repeats: y₁ is critiqued to produce fb₁, which refines to y₂, and so on until a stopping condition is met.

Model feedback versus external checks

Model feedback (self-refinement): the same model critiques and revises its own output. No extra tools needed; works for open-ended quality (dialogue, tone, code style). Unreliable for verifiable correctness — the model often cannot detect its own errors, and can confidently justify a wrong answer. External checks (tool-interactive critique): the model’s output is verified by an external tool — a search engine, a code interpreter, a calculator, a Perspective API-style scorer — and the critique comes from the tool’s result. More dependable for factual and executable correctness, but adds tool-call latency and cost per iteration.

2. Model self-feedback (Self-Refine)

Self-Refine (Madaan et al., 2023) uses a single model as generator, critic, and refiner, guided by three few-shot prompts (p_gen, p_fb, p_refine), with no additional training:

• y₀ = M(p_gen || x) — initial generation
• fb_t = M(p_fb || x || y_t) — self-feedback on the current draft
• y_{t+1} = M(p_refine || x || y₀ || fb₀ || ... || y_t || fb_t) — revision conditioned on the full history

The loop runs up to a maximum iteration count, or stops when the model emits a stop indicator.

What it is good for

Self-Refine improved over one-shot generation on open-ended and quality tasks in the paper’s evaluation (GPT-4 base):

Task	GPT-4 base	GPT-4 + Self-Refine	Change
Code optimization	27.3%	36.0%	+8.7 pts
Dialogue response (preference)	25.4%	74.6%	+49.2 pts
Code readability	27.4%	56.2%	+28.8 pts
Constrained generation	15.0%	45.0%	+30.0 pts

For math reasoning, Self-Refine barely moved GPT-4 (92.9% → 93.1%, +0.2 pts). The paper’s explanation: “a consistent-looking reasoning chain can deceive LLMs to think that ‘everything looks good’” — ChatGPT’s self-feedback was “everything looks good” for 94% of math instances. When the paper gave oracle (externally verified) feedback on math, gains were much larger (5%+). This is the core lesson: self-feedback is only as good as the model’s ability to spot its own errors, and on verifiable tasks that ability is weak.

Failure mode

In the paper’s qualitative failure analysis (35 failure cases), most problems were bad feedback, not bad revision: 33% mislocated the error, 61% suggested an inappropriate fix, and only 6% were the refiner mis-implementing good feedback. Bad critique reliably produces bad revision.

Do not assume self-correction means improvement

A model revising its own output is not the same as a model correcting its own output. On verifiable tasks, self-feedback can confidently ratify a wrong answer or “fix” a right one into a wrong one.

3. External tool checks (CRITIC)

CRITIC (Gou et al., 2023) verifies output against external tools and revises from the tool’s critique:

• Choose a tool appropriate to the task (search engine for facts, code interpreter/Python for math programs, Perspective API for toxicity).
• Verify y_t through the tool to obtain a critique c_t.
• If the critique indicates y_t is satisfactory, stop. Otherwise revise: y_{t+1} = Correct(x, y_t, c_t).

What it is good for

Factual correctness and executability:

Task	Tool	Result
Free-form QA (ChatGPT)	Search engine	F1 54.6 → 74.9 (+20.3 pts)
GSM8k math programs (ChatGPT)	Python interpreter	exact match 72.5% → 78.2% (+5.7 pts)
Toxicity reduction	Perspective API	max toxicity 0.325 → 0.173

The critical control: self-check vs tool-check

CRITIC includes a “w/o Tool” ablation where the model critiques itself instead of calling a tool. The result is the decisive evidence for this lesson:

• QA: model-only critiques contributed marginally (−0.03 / +2.33 F1) and could fall short of the initial output.
• GSM8k (text-davinci-003): model-only correction decreased exact match from 70.1% to 68.3% (−1.8 pts).
• GSM8k on Codex: Self-Refine (model-only) dropped exact match from 71.3% to 26.7% (−44.6 pts) — revision made the answer dramatically worse.

On truthfulness/self-verification specifically, models are poor judges of their own answers: CRITIC’s self-eval baseline scored only 54% accuracy on HotpotQA, worse than an “always say correct” baseline on TriviaQA.

4. Worked cases

4.1 Case where revision helps (model feedback)

Task: Optimize a slow sum function. Initial code:

def sum(n):
    res = 0
    for i in range(n + 1):
        res += i
    return res

Self-feedback: “This code is slow as it uses brute force. A better approach is to use the formula … (n(n+1))/2.” Revised code:

def sum_faster(n):
    return (n * (n + 1)) // 2

The feedback is specific (names the for loop) and actionable (suggests the formula), and the revision is verifiably better (O(1) vs O(n)). This is the Self-Refine code-optimization pattern, and it is why code optimization improved +8.7 pts with GPT-4. This case is illustrative of the paper’s Figure 2, not a measured run in this lesson.

4.2 Case where revision makes the answer worse (external check absent)

Task: GSM8k math word problem, text-davinci-003, program-of-thought baseline exact match 70.1%.

Without external feedback (model-only correction): exact match drops to 68.3% (−1.8 pts). The model’s self-critique suggested a change that broke a correct program.

More extreme (Self-Refine on Codex, model-only): exact match drops from 71.3% to 26.7% (−44.6 pts). The revision was far worse than the initial generation.

With external feedback (CRITIC, Python interpreter): exact match rises to 78.2% (+5.7 pts) — the tool’s execution feedback (error messages / results) gave a reliable correction signal.

The same “revise” operation helped with a trustworthy signal and hurt with an untrustworthy one. The signal source matters more than the revision mechanics.

5. Stopping policy and cost

Stopping policy

A defensible stopping policy combines three conditions; the loop stops when any is met:

1. Iteration cap: hard maximum (e.g., 4 iterations, matching Self-Refine’s evaluation; CRITIC used n=3 for QA, n=4 for math). Prevents unbounded cost and latency.
2. Stop indicator: the critique explicitly says the output is satisfactory / no change needed. Self-Refine extracts this from feedback; CRITIC checks whether the tool critique indicates satisfaction.
3. Unchanged output: two consecutive revisions produce the same answer (CRITIC’s early-stop rule). Continuing past this point only adds cost.

Cost model

Each iteration generates roughly two model passes (critique + refine), so:

• Token spend per iteration ≈ 2 × average output length (all generated tokens are billed, including discarded intermediate drafts).
• External-check iteration adds tool-call latency and API cost (search, interpreter, Perspective API).
• Latency is roughly sequential: generate → wait for tool → revise. Parallelizing candidates helps only if you generate multiple drafts up front (that is best-of-N / self-consistency territory, not refinement).

Rule of thumb: most of the benefit comes in the first 1–2 iterations; marginal returns diminish after that (Self-Refine Figure 4; CRITIC §4.5). Cap early.

6. Exit task: checkable exercise

Pick one task with a verifiable answer (e.g., GSM8k math or a fact-checkable QA) and one fixed model checkpoint. Do not edit chapters 21–22 or the curriculum; this lesson stands alone.

1. State the setup. Task, model, iteration cap (e.g., 3), stop indicator rule, unchanged-output rule, external tool (e.g., Python interpreter or search), and the independent correctness verifier.
2. Baseline. Measure one-shot accuracy on a fixed set of N examples.
3. Model-only refinement. Run generate–critique–revise with the model as its own critic (no tool). Measure accuracy after each iteration, parse rate, token spend, and wall-clock latency. Label results measured.
4. External-check refinement. Run the same loop with the external tool providing the critique. Measure the same metrics.
5. Decide. Adopt external-check refinement only if measured accuracy exceeds model-only refinement at the same iteration cap and the stop indicator / unchanged-output rule actually triggered (you are not just burning the cap every time). Adopt model-only refinement only for non-verifiable quality tasks (tone, style) where no external check exists. Otherwise reject.
6. Verification gaps. List at least three unmeasured items (e.g., tool-call failure rate, critique quality distribution, accuracy at iteration cap vs. earlier stop, cost per corrected example).

Any number without a measurement source is calculated or illustrative; do not claim improvements you have not reproduced.

Exercise: model feedback vs external check

1. 01State a stopping policy for a generate–critique–revise loop on a GSM8k-style task: iteration cap, stop indicator, and unchanged-output rule. Explain why each condition matters.

   Show a hint

   Cap at 3–4 iterations; stop when the tool critique says satisfactory or when two consecutive revisions match.

2. 02Given a baseline exact match of 70.1% and a model-only refinement result of 68.3% on the same set, state the change in percentage points and explain why this does not justify adopting model-only refinement.

   Show a hint

   Change = 68.3 − 70.1 = −1.8 pts. Revision made answers worse; adopt external checks instead.

3. 03Estimate the token spend for 3 iterations on outputs averaging 200 tokens, and state what cost is incurred even for discarded intermediate drafts.

   Show a hint

   3 iterations × 2 passes × 200 tokens = 1,200 tokens; every generated token is billed.

Related lessons

• Lesson 21 (how-reasoning-models-work): reasoning models emit intermediate tokens; refinement adds more tokens on top.
• Lesson 22 (evaluating-reasoning): baselines, answer normalization, extraction, and parsing failures — the independent verifier used here.
• Lesson 23 (inference-time-scaling): repeated sampling, best-of-N, and self-consistency — alternative ways to spend inference-time compute.

Related lessons

• LLM Internals & Architecture · Reasoning models and test-time computationHow Reasoning Models Work: Training Paths, Traces, and Cost16 min
• LLM Internals & Architecture · Reasoning models and test-time computationEvaluating Reasoning: Baselines, Verifiers, and Parsing Failures12 min
• LLM Internals & Architecture · Reasoning models and test-time computationInference-Time Scaling: Sampling, Best-of-N, and Self-Consistency13 min

Sources and claim-to-source mapping

Source	Exact section	What it supports	Scope and limits
Madaan et al. (2023), “Self-Refine: Iterative Refinement with Self-Feedback” (arXiv:2303.17651)	§2 (algorithm, Eqns. 1–4), §3.1 (base LLMs), §3.2/§3.3 (results Table 1), §4 (feedback quality, iterations), qualitative failure analysis	Generate→feedback→refine with same model, few-shot prompts, max 4 iterations; math near-flat (+0.2 GPT-4), “everything looks good” 94% of instances, oracle feedback 5%+ math gains; failure analysis 33%/61%/6%; code optimization +8.7 GPT-4; dialogue +49.2; preferred over k=4 raw samples; Vicuna-13B too weak	GPT-3.5/4, Codex; specific tasks and prompts; gains not universal; paper does not release model weights
Gou et al. (2023), “CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing” (arXiv:2305.11738)	§3 (algorithm, verify→correct), §4.1 (QA results), §4.2 (GSM8k results), §4.4 (self-verification reliability), §4.5 (ablations)	External tools improve factual/math correctness; model-only (“w/o Tool”) correction marginal or harmful (−1.8 GSM8k text-davinci-003; Self-Refine on Codex −44.6); self-eval 54% HotpotQA; stop on unchanged output for two consecutive corrections; diminishing returns after 2–3 rounds	GPT-3.5 variants; specific QA/GSM8k/toxicity sets; tool APIs required; not all tasks suit external tools

Verification gaps stated:

• All arithmetic in this lesson (percentage-point changes, token-spend estimate) is calculated from stated figures; it is illustrative unless measured by the reader.
• The worked code case (4.1) is illustrative of Self-Refine’s Figure 2, not a measured run in this lesson.
• Primary-source results are specific to the cited models, datasets, and tools; they do not transfer to your task without measurement.
• Model-only refinement can degrade performance; the lesson does not claim self-correction is universally beneficial.