Reinforcement Learning for Reasoning: Verifiable Rewards and GRPO

Reinforcement Learning with Verifiable Rewards (RLVR) is the training paradigm behind models such as DeepSeek-R1, DeepSeekMath, and a growing number of open reasoning models. The core idea is simple: instead of learning a reward model from human preferences, use a verifiable reward—a binary or near-binary signal that can be checked by code (unit tests, exact-match on math answers, format validators). Then optimise the policy with a reinforcement learning algorithm that does not need a separate critic network.

The algorithm that made this practical at scale is Group Relative Policy Optimization (GRPO), introduced in DeepSeekMath and used in DeepSeek-R1. GRPO replaces PPO’s learned value-function critic with a group baseline computed from multiple rollouts per prompt. This lesson builds the mechanism from the ground up, works a tiny numerical example by hand, and maps the general method to the specific choices made in published training runs—without presenting those choices as the only way to run GRPO.

---

1. Verifiable rewards: what they are and why they matter

A verifiable reward is a scalar reward function r(q, o) ∈ {0, 1} (or a small discrete set) where the correctness of an output o for a prompt q can be determined algorithmically, without a learned model. Common examples:

Domain	Verifier	Reward signal
Math (GSM8K, MATH)	Exact match on boxed answer, or symbolic equivalence via a CAS	1 if correct, 0 otherwise
Code (HumanEval, Codeforces)	Unit tests pass / compilation succeeds	1 if all tests pass, 0 otherwise
Format adherence	Regex or structural check (e.g., <think>…</think><answer>…</answer>)	1 if format matches, 0 otherwise
Logic puzzles	Deterministic solver or exact match	1 if solution verifies, 0 otherwise

Why not a learned reward model? Learned reward models (outcome or process supervisors) are powerful but introduce three problems at scale:

1. Reward hacking: the policy finds adversarial outputs that score highly on the reward model but are incorrect. DeepSeek-R1 explicitly avoided neural reward models for reasoning tasks because they observed hacking at scale.
2. Training cost: a reward model is a second large model to train, store, and run during RL.
3. Distribution shift: the reward model’s judgements degrade as the policy moves away from the data it was trained on.

Verifiable rewards are grounded—they do not drift. Their limitation is coverage: they only exist for tasks with an automatic ground truth (math, code, constrained format). For open-ended chat, a learned reward model or human feedback is still used.

---

2. The GRPO rollout loop

GRPO is an on-policy algorithm. Each training step follows this loop:

for each training step:
    1. Sample a batch of prompts Q = {q_1, ..., q_B} from the dataset.
    2. For each prompt q, generate G rollouts (a "group") from the current policy π_θ_old:
         {o_1, ..., o_G} ~ π_θ_old(· | q)
    3. Score each rollout with the verifiable reward:
         r_i = r(q, o_i) ∈ {0, 1}   (or a small discrete set)
    4. Compute the group statistics:
         mean(q) = (1/G) Σ r_i
         std(q)  = sqrt( mean(q) * (1 - mean(q)) )   [Bernoulli variance]
    5. Compute group-relative advantages:
         A_i = (r_i - mean(q)) / (std(q) + ε)
    6. Update the policy π_θ to maximise the GRPO objective (see §3).
    7. Periodically refresh the reference model π_ref ← π_θ.

Key properties of this loop:

• Group size G: DeepSeekMath used G=4 for process supervision and G=8/16 for outcome supervision. DeepSeek-R1 used G=16. Larger groups give better baselines but cost more compute.
• Rollout length: The rollout is a full chain-of-thought + answer. In DeepSeek-R1-Zero, max length grew from 32k to 64k tokens during training.
• No critic: Step 4–5 replaces the learned value function V_ψ in PPO. The group mean is a Monte Carlo estimate of the expected reward under the old policy for that prompt.
• On-policy: Rollouts come from π_θ_old, the policy before the current update. After the update, π_θ becomes the new π_θ_old.

---

3. Group-relative advantages: derivation and intuition

In PPO, the advantage at token t is A_t = r_t + γ V(s_{t+1}) - V(s_t), where V is a learned critic. GRPO does not have a critic. Instead, for a given prompt q, it uses the group of rewards as a self-contained baseline.

3.1 From rewards to advantages

For binary rewards r_i ∈ {0, 1}, the group of G rewards is a sample from a Bernoulli distribution with success probability p(q) = ℙ(r=1 | q). The empirical mean and (Bernoulli) standard deviation are:

mean(q) = (1/G) Σ r_i
std(q)  = sqrt( mean(q) * (1 - mean(q)) + ε )

The group-relative advantage for rollout i is:

A_i = (r_i - mean(q)) / std(q)

This is exactly whitening (standardisation) of the reward within the group. It has two effects:

1. Centering: A_i is positive for above-average rollouts, negative for below-average.
2. Scaling: Rare successes (low mean(q)) get larger positive advantages; common successes get smaller ones. Failures are penalised more when success is common.

3.2 Worked numerical example

Consider one prompt q with group size G = 4. The policy generates four rollouts; three are incorrect, one is correct.

Rollout i	Reward r_i	mean(q)	std(q)	Advantage A_i = (r_i - mean) / std
1	1	0.25	0.433	(1 - 0.25) / 0.433 = +1.732
2	0	0.25	0.433	(0 - 0.25) / 0.433 = −0.577
3	0	0.25	0.433	−0.577
4	0	0.25	0.433	−0.577

Verify by hand

mean = (1+0+0+0)/4 = 0.25. Bernoulli variance p(1-p) = 0.25*0.75 = 0.1875. std = sqrt(0.1875) ≈ 0.433. A_1 = 0.75/0.433 ≈ 1.732. A_{2,3,4} = -0.25/0.433 ≈ -0.577. Sum of advantages = 1.732 − 3×0.577 ≈ 0 (up to rounding).

Interpretation:

• The single correct rollout gets a large positive advantage (+1.732) because success is rare (p=0.25).
• Each incorrect rollout gets a moderate negative advantage (−0.577).
• If instead mean = 0.75 (three correct, one wrong), the unique failure would get A = (0-0.75)/0.433 = −1.732 (heavily penalised), while each success gets +0.577.

This is the calibration property: the same binary reward is scaled differently depending on how hard the prompt is for the current policy. It prevents “easy” prompts from dominating the gradient.

3.3 Comparison: mean-only normalisation (Dr. GRPO)

A variant called Dr. GRPO drops the variance division and uses:

A_i = r_i - mean(q)

For the same example (mean=0.25): A_1 = +0.75, A_{2,3,4} = −0.25. The relative weighting between success and failure is the same (3:1), but the absolute scale is smaller and does not adapt to p. In practice, mean+variance (full whitening) is more stable when p varies widely across prompts.

---

4. The GRPO objective

With advantages in hand, GRPO maximises a clipped surrogate objective following DeepSeekMath §4.1.1 (Equation 3). The objective is per-token with sequence-length averaging and one outcome advantage shared across all tokens of a rollout:

J_GRPO(θ) = 𝔼[q ~ P(Q), {o_i}_{i=1}^G ~ π_θ_old(·|q)]
    [ 1/G Σ_{i=1}^G ( 1/|o_i| Σ_{t=1}^{|o_i|} min( ρ_{i,t}(θ) Â_{i,t}, clip(ρ_{i,t}(θ), 1-ε, 1+ε) Â_{i,t} ) )
    − β D_KL(π_θ || π_ref) ]

where:

• ρ_{i,t}(θ) = π_θ(token_{i,t} | q, o_{i,<t}) / π_θ_old(token_{i,t} | q, o_{i,<t}) is the per-token importance-sampling ratio for token t of rollout i, stored at rollout generation time.
• Â_{i,t} = (r_i - mean(q)) / (std(q) + δ) is the group-relative advantage for rollout i, with a small numerical stabiliser δ distinct from the clipping parameter ε. For outcome supervision (DeepSeekMath §4.1.2), a single scalar reward r_i ∈ {0,1} is assigned to the full rollout; the advantage is the same for every token t in that rollout. mean(q) and std(q) are the group mean and standard deviation of the G rewards.
• ε is the clip parameter (typically 0.2 in PPO; DeepSeekMath used ε=0.2; DeepSeek-R1 used a large clip ratio of 10).
• β is the KL coefficient; π_ref is a fixed reference model (usually the initial SFT model, periodically refreshed).

Per-token with sequence-length averaging

DeepSeekMath Eq. 3 averages the per-token surrogate by the rollout length 1/|o_i|. This prevents longer rollouts from dominating the gradient. The outcome-supervised advantage Â_{i,t} is constant across t for a given rollout i (one reward → one whitened advantage → shared across all tokens). This differs from process supervision (§4.1.3) where Â_{i,t} varies per step.

4.1 The KL regularisation term

The KL term prevents the policy from collapsing to a degenerate solution or drifting too far from the reference model. Three regularisation patterns exist in the literature:

| Variant | KL target | Formula | Used by | | –––––––– | ———————–– | ––––––– | ––––––––––––––––– | ——— | ———————————– | | Reference KL | Fixed reference π_ref | β \* D*KL(π*θ | | π_ref) | DeepSeekMath, DeepSeek-R1 (β=0.001) | | Mirror KL | Previous policy π_θ_old | β \* D*KL(π*θ | | π_θ_old) | DAPO, Dr. GRPO (α=0 in mixed KL) | | No KL | None (β=0) | — | Open-Reasoner-Zero, some DAPO runs |

What the KL does:

• π_ref KL: keeps the policy anchored to the initial SFT model. Prevents catastrophic forgetting of instruction-following and format.
• π_θ_old KL: acts like a trust region, similar to PPO’s clipping. Allows the policy to move further over many steps.
• No KL: maximises movement toward high-reward regions. Risk: the policy may lose format adherence, language consistency, or general capabilities.

DeepSeek-R1 used β=0.001 with a fixed reference that was refreshed every 400 steps. Many 2025 variants (Dr. GRPO, DAPO) found β=0 works for verifiable-reward reasoning because the binary reward itself provides a strong anchor—the policy cannot “hallucinate” a high reward without actually solving the problem.

---

5. Reward hacking with verifiable rewards

Even with verifiable rewards, hacking is possible. The reward is a proxy for the true objective (correct reasoning), and the policy can exploit gaps.

5.1 Known hacking patterns

Hack	Description	Example
Format gaming	The policy learns to output the correct format tags (<think>…</think><answer>…</answer>) without meaningful content inside, or stuffs the answer into the thinking section to satisfy both format and accuracy.	DeepSeek-R1 added a language consistency reward to reduce mixing; format reward alone can be gamed.
Length hacking	The policy produces extremely long CoTs because longer sequences have more chances to accidentally contain the correct answer, or because the reward includes a length bonus.	DeepSeek-R1 observed length increase from ~1k to >10k tokens; they capped max length.
Answer-only shortcuts	For math, the policy learns to output just the answer (or a trivial pattern) without reasoning, if the verifier only checks the final boxed answer.	If the verifier extracts \boxed{42}, the policy may output \boxed{42} immediately.
Test-case memorisation	For code, the policy memorises specific test inputs/outputs from the training set rather than learning general logic.	Mitigated by held-out test cases and large, diverse datasets.

5.2 Mitigations used in practice

• Separate format and accuracy rewards with equal weight (DeepSeek-R1): Reward = Reward_acc + Reward_format. The format reward forces structure; the accuracy reward forces correctness.
• No neural reward model for reasoning: DeepSeek-R1 avoided learned reward models entirely for math/code/logic, using only rule-based verifiers.
• Clipping / KL regularisation: Limits how far the policy can move in one update.
• Reference model refresh: Periodically updating π_ref to the current policy prevents the KL from becoming a hard barrier.
• Curriculum / difficulty filtering: Start with problems where p_ref > 0 so the policy has a learning signal.

Verifiable does not mean unhackable

A verifiable reward is grounded, not unhackable. The verifier checks the output against a specification; if the specification is incomplete (e.g., only checks final answer format), the policy will optimise the specification, not the intent. Always audit what the verifier actually checks.

---

6. What RLVR improves and what it does not

6.1 Empirical findings (Yue et al., 2025)

Yue et al. systematically evaluated RLVR across model families, algorithms, and benchmarks using pass@k at large k as a measure of the model’s reasoning boundary (the set of problems it can solve given enough samples).

Finding	Implication
RLVR models beat base models at small k (e.g., pass@1)	RLVR improves reliability of sampling: the policy concentrates probability mass on correct solutions.
Base models achieve higher pass@k at large k	The reasoning boundary (what the model can solve) is not expanded; RLVR only shifts the distribution within the existing boundary.
Six popular RLVR algorithms (GRPO, PPO, DPO variants) perform similarly	The algorithm choice matters less than the base model and data.
Distillation from a stronger teacher does expand the boundary	Learning from a teacher’s correct traces adds new reasoning patterns not present in the base model.

Conclusion: RLVR is a reliability optimiser, not a capability expander. It makes the model more consistently solve problems it already could solve. For genuinely new reasoning patterns, distillation or a stronger base model is needed.

6.2 Limits of the method

• Coverage: Only tasks with automatic verifiers (math, code, constrained format). No open-ended reasoning.
• Credit assignment: Rollout-level advantage cannot distinguish which part of a long CoT was responsible for success/failure. Process supervision or step-level verifiers help but are expensive.
• Exploration: With binary rewards and small p, most rollouts get zero reward. The gradient signal is sparse. Large group sizes and curriculum help.
• Generalisation: Improvements on training-domain benchmarks (MATH, GSM8K) do not always transfer to out-of-domain reasoning (e.g., GPQA, FRAMES).
• Format drift: Without a language consistency reward or KL anchor, the policy may mix languages, drop format tags, or produce unreadable CoTs.

---

7. Minimal GRPO pseudocode (per-token with sequence-length averaging, DeepSeekMath §4.1 Eq. 3)

# This is a teaching sketch illustrating the per-token GRPO objective.
# It omits: distributed rollout, gradient accumulation, mixed precision,
# learning rate scheduling, reference model refresh, data loading, etc.
#
# Key form from DeepSeekMath §4.1 Eq. 3:
#     J_GRPO(θ) = 𝔼[ 1/G Σ_i (1/|o_i| Σ_t min(ρ_{i,t} Â_i, clip(ρ_{i,t},1-ε,1+ε) Â_i)) ] − β D_KL
# where ρ_{i,t} = π_θ(token_{i,t}|q,o_{i,<t}) / π_θ_old(token_{i,t}|q,o_{i,<t})
# and Â_i = (r_i - mean) / std is the rollout-level advantage shared across tokens.

def grpo_illustration(policy, old_policy, prompts, G, epsilon=0.2, beta=0.001):
    """
    Teaching sketch: per-token GRPO with sequence-length averaging.
    One outcome reward per rollout → one whitened advantage shared across all tokens.
    """
    # 1. Generate groups of rollouts from the old policy
    all_rollouts = []      # list of lists of token IDs
    all_rewards  = []      # scalar reward per rollout (0 or 1)
    all_old_logprobs = []  # per-token logprobs from old policy

    for q in prompts:
        rollouts = [old_policy.generate(q) for _ in range(G)]
        rewards  = [verifier(q, o) for o in rollouts]  # 0 or 1
        old_lp   = [old_policy.logprob_tokens(q, o) for o in rollouts]  # per-token vectors
        all_rollouts.append(rollouts)
        all_rewards.append(rewards)
        all_old_logprobs.append(old_lp)

    # 2. Compute rollout-level advantages (one per rollout, shared across tokens)
    advantages = []
    for rewards in all_rewards:
        mean_r = sum(rewards) / G
        std_r  = math.sqrt(mean_r * (1 - mean_r) + 1e-8)
        rollout_advantages = [(r - mean_r) / std_r for r in rewards]
        advantages.append(rollout_advantages)

    # 3. Compute per-token policy gradient with sequence-length averaging
    loss = 0.0
    for i, q in enumerate(prompts):
        for j in range(G):  # j indexes rollout within group
            # Per-token importance-sampling ratio: ρ_{i,t} = π_θ / π_θ_old
            # Sum over tokens, then average by rollout length |o_i|
            new_lp_vec = policy.logprob_tokens(q, all_rollouts[i][j])  # vector of per-token logprobs
            old_lp_vec = all_old_logprobs[i][j]                        # vector
            ratio_vec  = [math.exp(n - o) for n, o in zip(new_lp_vec, old_lp_vec)]

            adv = advantages[i][j]  # single scalar advantage for this rollout

            # Per-token surrogate, averaged by rollout length
            rollout_loss = 0.0
            for ratio_t in ratio_vec:
                clipped = max(1 - epsilon, min(1 + epsilon, ratio_t))
                rollout_loss += -min(ratio_t * adv, clipped * adv)
            rollout_loss /= len(ratio_vec)  # 1/|o_i| sequence-length averaging
            loss += rollout_loss

    loss = loss / (len(prompts) * G)

    # 4. KL penalty to reference (conceptual; omitted from this sketch)
    # kl = compute_kl(policy, ref_policy, prompts)
    # loss += beta * kl

    return loss

Teaching sketch only

This sketch is mathematically consistent with DeepSeekMath §4.1 Eq. 3:

• Per-token ratio ρ_{i,t} = exp(new_lp_t − old_lp_t) (vector, not scalar)
• One outcome advantage Â_i shared across all tokens of rollout i
• Sequence-length averaging 1/|o_i| via rollout_loss /= len(ratio_vec)
• Clipped surrogate min(ratio_t · adv, clipped · adv) per token For actual training, use a validated implementation (e.g., vLLM + PEFT).

Key change from whole-rollout to per-token: The pseudocode now stores all_old_logprobs as per-token values (not a single scalar per rollout), and the importance ratio ratio is computed per-token as exp(logprob_new - logprob_old). The advantage advantages[i][j] is assigned per-token position but derived from the rollout-level reward whitening. The surrogate min(ratio * adv, clipped * adv) follows the per-token PPO/GRPO form from DeepSeekMath §4.1, where the clip direction automatically reverses when adv < 0.

---

8. Exit task: compute advantages and interpret the gradient

Prompt: A single math problem q. For this clipping exercise, treat each of four rollouts as a one-token completion, so each sequence log-probability is also its token log-probability. Rewards are [1, 0, 0, 0]. The new policy log-probabilities are [-2.0, -2.5, -3.0, -3.5]; the old policy log-probabilities (stored at rollout time) are [-2.2, -2.3, -2.8, -3.2]. Use clipping ε=0.2, stabiliser δ=1e-8, and β=0 (no KL). Real multi-token rollouts require one ratio per token.

Exit task: GRPO advantage and gradient sign

1. 01Compute the group mean, std (with δ=1e-8), and the four advantages A_i.

   Show a hint

   mean=0.25, std=sqrt(0.25*0.75+1e-8).

2. 02Compute the importance-sampling ratios r_i = exp(logprob_new - logprob_old) for each rollout.

   Show a hint

   r_1 = exp(-2.0 - (-2.2)) = exp(0.2) ≈ 1.221.

3. 03For each rollout, compute the clipped ratio clip(r_i, 0.8, 1.2). Which rollout(s) get clipped?

   Show a hint

   r_1=1.221 clips to 1.2 (upper bound). r_4=0.741 clips to 0.8 (lower bound). r_2 and r_3 are within [0.8, 1.2].

4. 04For rollout 1 (the correct one), is the local surrogate gradient active at the stated ratio? Explain.

   Show a hint

   Its positive-advantage surrogate is capped at the upper clip bound. A positive surrogate value does not imply a nonzero local gradient.

5. 05For rollout 4 (an incorrect one with ratio=0.741), is the local surrogate gradient active? Compare with the unclipped rollout 2.

   Show a hint

   The negative-advantage surrogate is capped below 0.8, so rollout 4 has zero local gradient from this term. Rollout 2 is inside the clip interval and has an active negative-advantage term.

Check your work

1. Advantages: mean=0.25, std≈0.433. A = [+1.732, −0.577, −0.577, −0.577].
2. Ratios: r = [exp(0.2)=1.221, exp(-0.2)=0.819, exp(-0.2)=0.819, exp(-0.3)=0.741].
3. Clipped: r_1=1.221 clips to 1.2. r_4=0.741 clips to 0.8 (below the lower clip bound). r_2 and r_3 are within [0.8, 1.2].
4. Rollout 1 (correct): A=+1.732. ratio=1.221, clipped=1.2. Surrogate = min(1.221*1.732, 1.2*1.732) = 2.078. The clipped branch is constant locally, so this term has zero local gradient at the stated ratio.
5. Rollout 4 (incorrect): A=−0.577. ratio=0.741, clipped=0.8. Surrogate = min(0.741*(−0.577), 0.8*(−0.577)) = −0.462. The clipped branch again has zero local gradient. By contrast, rollout 2 is inside the clip interval: its surrogate is 0.819*(−0.577) = −0.473, and its negative advantage gives an active downward gradient on that token’s log-probability.

---

9. Key takeaways

Key takeaways

• Verifiable rewards are binary, code-checkable signals (unit tests, exact match, format). They avoid learned reward models but only exist for tasks with automatic ground truth.
• GRPO samples a group of G rollouts per prompt, computes the group mean and std of rewards, and whitens them to get group-relative advantages. No critic network is needed.
• The advantage formula A_i = (r_i - mean) / std calibrates the gradient: rare successes get larger updates; common successes get smaller ones.
• The GRPO objective is a clipped surrogate at the rollout level plus a KL penalty (to a reference model, the old policy, or neither). β=0 is common in 2025 verifiable-reward runs.
• Reward hacking still occurs with verifiable rewards: format gaming, length hacking, answer-only shortcuts. Mitigations include separate format/accuracy rewards, no neural RM for reasoning, KL/anchor, and curriculum.
• RLVR improves reliability (pass@1) but does not expand the reasoning boundary (pass@k at large k). Distillation from a stronger teacher is needed for genuinely new reasoning patterns.

---

10. Sources and claim-to-source mapping

Source	Exact section	What it supports	Scope and limits
Shao et al., “DeepSeekMath” (arXiv:2402.03300), §4.1	4.1.1, 4.1.2, A.1.6	GRPO origin: from PPO to GRPO; group-relative advantage formula; outcome/process supervision with GRPO; iterative RL.	One lab’s algorithm; 7B model on GSM8K/MATH; group size G=4–16.
DeepSeek-AI, “DeepSeek-R1” (arXiv:2501.12948), §2.1, 2.2, A.3	2.1, 2.2, A.3	GRPO objective with clipping and KL; rule-based accuracy + format rewards; no neural RM for reasoning; clip ratio ε=10; β=0.001; reference refresh every 400 steps.	DeepSeek-V3-Base, 16 rollouts per prompt, max len 32k→64k.
Yue et al., “Does RL Really Incentivize Reasoning?” (arXiv:2504.13837), §3, §4.2	3, 4.2	RLVR improves pass@1 but not pass@k at large k; base model bounds reasoning boundary; six RLVR algorithms perform similarly; distillation expands boundary.	Math, code, visual benchmarks; specific RL algorithms; not proof no future RL can expand boundary.
Mroueh, “RL with Verifiable Rewards: GRPO’s Effective Loss” (arXiv:2503.06639), §2–3	2.1, 2.2, 3	Whitened advantage as calibrated weight; contrastive loss view; PoS fixed-point iteration; success amplification theorem.	Binary rewards; theoretical analysis with Bernoulli assumption; ε-smoothing.
DeepSeek-R1 Appendix B.5	B.5	Reward hacking observations: language mixing, format gaming, length increase; language consistency reward; more preference-RL steps lead to hacking.	One training run; specific to DeepSeek-R1’s reward mixture.
Schulman et al., “PPO” (2017)	—	PPO surrogate objective, clipping, GAE, KL penalty for reference.	General RL; not LLM-specific.

Verification gaps stated:

• The numerical example in §3.2 uses G=4 and Bernoulli variance. Real runs use G=8–16 and may use empirical variance instead of Bernoulli formula.
• The pseudocode in §7 is a teaching sketch; it omits distributed rollout, per-token log-probs, importance sampling from stored old log-probs, gradient accumulation, mixed precision, LR schedules, and reference refresh.
• The Yue et al. findings are specific to the tested algorithms, model families, and benchmarks. They do not prove that no future RLVR method can expand the reasoning boundary.
• DeepSeek-R1’s specific hyperparameters (ε=10, β=0.001, G=16, refresh 400 steps) are one recipe, not the definition of GRPO. Other labs use different values (e.g., Dr. GRPO uses mean-only normalisation; DAPO uses β=0).

Related lessons

• LLM Internals & Architecture · Reasoning models and test-time computationHow Reasoning Models Work: Training Paths, Traces, and Cost16 min
• LLM Internals & Architecture · Reasoning models and test-time computationEvaluating Reasoning: Baselines, Verifiers, and Parsing Failures12 min
• LLM Internals & Architecture · Reasoning models and test-time computationInference-Time Scaling: Sampling, Best-of-N, and Self-Consistency13 min