Inference-Time Scaling: Sampling, Best-of-N, and Self-Consistency

Spending more inference-time compute can improve an answer without changing model weights. This lesson compares three standard ways to do it — repeated sampling, best-of-N selection, and self-consistency — under the same token, latency, and selection budgets, so the comparison is fair and the tradeoffs are explicit.

The one-sentence version: Repeated sampling, best-of-N, and self-consistency all buy correctness with extra generated tokens, but they spend those tokens differently: repeated sampling hedges across attempts, best-of-N picks the single highest-scored candidate under a reward model, and self-consistency picks the answer that most sampled reasoning paths agree on. None guarantees improvement; each has identifiable failure modes.

1. A fixed-budget comparison framework

Before any method is compared, fix what is being held constant. Without this, a method that simply spends more tokens will always look better, and the comparison tells you nothing about the method itself.

Budget dimension	What to hold constant	Why
Token budget	Maximum generated tokens per prompt (e.g., 2,000 output tokens)	Repeated sampling, best-of-N, and self-consistency each consume tokens per candidate; without a shared token ceiling, a method that generates more candidates wins by spending more, not by being better.
Latency budget	Wall-clock time or per-request latency ceiling under the same serving stack	Best-of-N and self-consistency require N full decodes before selection; latency scales with N and with candidate length.
Selection budget	Number of candidates N that can be scored/selected	Best-of-N scores all N under a reward model; self-consistency parses and votes over all N answers. The parser, verifier, and reward model are part of the cost.
Prompt and model	Same checkpoint, same temperature/top-p, same chat template	The base policy produces the candidates; changing it confounds method with model.
Answer extraction	Same parser and validity check for every method	A method with a looser parser looks better because it accepts more outputs, not because it answers better.

Calculated versus measured. Everything in sections 2–4 is calculated from stated probability models or reported primary-source results; it is illustrative unless your own run measures it. Section 6 is the decision exercise: it tells you what to measure before adopting any method.

Rejected candidates still cost

Best-of-N and self-consistency generate N candidates and then discard all but one. Every generated token is billed and occupies KV-cache memory during its decode, even if it is never selected. Token spend is N × (average candidate length), not 1 × length.

2. The three methods

All three methods start from the same base policy π (your model at a fixed temperature) and sample N candidates for one prompt. They differ in what they do with those candidates.

2.1 Repeated sampling (sample once, keep everything)

Generate N independent candidates from π. Return all N candidates to the user or downstream system; there is no single selection.

• Selection mechanism: none at inference time. Selection is deferred to whoever consumes the candidates (a user, a verifier, a downstream pipeline).
• Token spend: N × average candidate length, all generated and returned.
• When defensible: When downstream consumers can evaluate or combine candidates themselves, or when you want to surface uncertainty rather than collapse it to one answer.
• Failure mode: N wrong candidates are still N wrong candidates. If the base policy’s answer mass on the correct answer is small, repeated sampling does not create correctness; it only exposes variance.

2.2 Best-of-N (generate N, score N, return the top)

Generate N candidates from π, score each with a reward model r̂, and return argmaxᵢ r̂(x, yᵢ).

• Selection mechanism: maximize a reward model score over N samples. Requires either a learned reward model (trained separately) or a rule-based verifier (e.g., unit tests, exact match) as r̂.
• Token spend: N × average candidate length, plus reward-model evaluation over all N (usually cheap relative to generation but not free).
• When defensible: You have a reliable reward model or verifier whose scores correlate with true correctness, and you can afford N decodes plus scoring.
• Failure mode: reward hacking. r̂ is an imperfect proxy for true correctness. As N grows, the maximum of r̂ is increasingly likely to be a candidate that r̂ overrates but is wrong under the true reward. Best-of-N can degrade in accuracy as N increases past a critical point (Huang et al., 2025, §3.1). The failure is not a bug in the code; it is the mathematical consequence of maximizing an imperfect proxy over many draws.

2.3 Self-consistency (generate N, majority-vote answers)

Generate N candidate reasoning paths from π under chain-of-thought prompting, parse the final answer from each, and return the answer with the most votes (plurality; tie broken by first encountered).

• Selection mechanism: majority/plurality vote over parsed final answers, not over tokens or reasoning paths. Correct but differently-reasoned paths reinforce each other; incorrect paths tend to scatter across different wrong answers (Wang et al., 2022, §2).
• Token spend: N × average candidate length. No separate reward model is required — the vote itself is free once parsing is done — but parsing must succeed for a candidate to cast a vote.
• When defensible: Tasks with discrete, verifiable answers (math word problems, multiple choice) where multiple reasoning paths lead to the same correct answer.
• Failure mode: shared blind spot. If the base policy has a systematic misconception, most sampled paths can agree on the same wrong answer, and self-consistency confidently returns it. It also fails when too few candidates parse to a valid answer.

Do not conflate the three

Repeated sampling returns candidates; best-of-N returns the top candidate under r̂; self-consistency returns the most frequent parsed answer. A method that works for multiple-choice math (self-consistency) is not the same as one that works for open-ended chat with a learned reward model (best-of-N). Choose by task and by what selection signal you trust.

3. Selection mechanism detail

The selection step is where each method can fail independently of generation quality.

Method	Selection input	Selection output	What can go wrong
Repeated sampling	None (deferred)	N candidates	All N wrong; consumer overwhelmed
Best-of-N	r̂(x, yᵢ) for each of N candidates	One candidate (max r̂)	Reward hacking: max r̂ ≠ max true correctness
Self-consistency	Parsed answer aᵢ from each of N candidates	Most frequent answer	Shared blind spot; too few parseable answers

Single-answer selection accuracy applies only to best-of-N and self-consistency, both of which return one answer. Repeated sampling returns all N candidates with no single selection, so its “accuracy” is not comparable on the same axis: the right metric for repeated sampling is P(at least one correct among N) = 1 − (1 − p)ᴺ (section 4.1), measured with an independent verifier that can identify which candidate is correct. Do not compare best-of-N’s single-answer accuracy directly against repeated sampling’s N-candidate output; they answer different questions.

Worked selection example (toy)

Prompt: “A hall has 12 rows of 4 chairs. 7 chairs are removed. How many remain?” Correct answer: 41.

Candidate	Reasoning (abbreviated)	Parsed answer	r̂ score	Self-consistency vote
1	12×4 = 50; 50 − 7 = 43	43	0.9	—
2	12×4 = 48; 48 − 7 = 41	41	0.6	41
3	12×4 = 48; 48 − 7 = 41	41	0.5	41
4	12×4 = 48; 48 − 2 = 46	46	0.8	—
5	12×4 = 48; 48 − 7 = 41	41	0.4	41

• Repeated sampling: returns all five; downstream must notice 41 appears three times.
• Best-of-N: returns candidate 1 (answer 43, r̂ = 0.9) — wrong, because r̂ preferred the confident miscalculation.
• Self-consistency: returns 41 (3 of 5 votes) — correct.

This is an illustrative toy. The reward model preferring the wrong answer is the reward-hacking failure in miniature; the agreement on 41 is the self-consistency success in miniature.

4. Worked arithmetic

All figures below are calculated from stated probability models, not measured from a model run. They show how to size budgets; they are not predictions.

4.1 Expected correct candidates

Let p be the probability that one sample from π produces the correct final answer (independently, approximately). For N samples:

• Expected number correct: N · p
• Probability at least one correct: 1 − (1 − p)ᴺ
• Probability all N wrong: (1 − p)ᴺ

Example: p = 0.25, N = 8.

• Expected correct: 8 × 0.25 = 2
• Probability at least one correct: 1 − 0.75⁸ ≈ 1 − 0.100 = 0.900
• Probability all wrong: 0.75⁸ ≈ 0.100

So even with a weak base policy (p = 0.25), 8 samples give a 90% chance that at least one candidate is correct — but finding it still requires a selection method that identifies the correct one. Repeated sampling alone does not surface it; best-of-N needs r̂ to rank it first; self-consistency needs correct answers to outnumber any single wrong answer.

4.2 Token spend under a fixed token budget

Token budget = B tokens total; average candidate length = L tokens.

• Maximum candidates: Nₘₐₓ = ⌊B / L⌋
• Actual token spend for N candidates: N × L
• Leftover tokens: B − N × L (wasted unless another prompt uses them)

Example: B = 2,000, L = 250 → Nₘₐₓ = 8. Spending on 6 candidates uses 1,500 tokens and leaves 500 idle.

Implication: best-of-N and self-consistency must choose N before knowing candidate lengths. If candidates run long, you get fewer of them; if short, more. The selection quality depends on the realized N, not the planned one.

4.3 Self-consistency majority probability (toy model)

Suppose each sample is correct with probability p, and whenever it is wrong it picks one of W wrong answers uniformly at random. The correct answer wins the plurality vote if its count exceeds every wrong answer’s count.

This has no closed form for general W, but the boundary case is informative:

• If p > 1 / (1 + W) — i.e., the correct answer is more likely than any single wrong answer — then as N grows, the correct answer’s expected plurality share eventually exceeds any wrong answer’s, and self-consistency’s win probability approaches 1.
• If p < 1 / (1 + W), the most likely wrong answer can dominate, and self-consistency converges to the wrong answer as N grows — more samples make it worse, not better.

This is the shared-blind-spot failure in probability form. It is a toy model; real answer distributions are not uniform over W wrong answers, and p is not known without measurement.

5. Failure cases

Failure	Method	Symptom	Cause	Check
All samples wrong	All	No candidate contains the answer	Base policy p too low for the task	Track parseable answers across N; if none match the key, no selection method can recover it
Reward hacking	Best-of-N	Accuracy rises then falls as N grows	r̂ overrates tail candidates	Measure accuracy vs N with an independent verifier, not r̂
Shared blind spot	Self-consistency	Most votes agree on a wrong answer	Systematic base-policy misconception	Inspect the losing candidates’ reasoning; if they share the same error, the vote is wrong
Parse failure	Best-of-N / self-consistency	Fewer than N candidates scored/voted	Answer extractor drops outputs	Log parse rate vs N; a low parse rate silently reduces effective N

Latency is not token count

N candidates decoded sequentially take roughly N × single-decode latency. Decoded in parallel on enough hardware, wall-clock approaches one decode plus selection overhead, but you pay for N × L token-generation compute regardless. Measure end-to-end latency under your serving stack and batch size; do not infer it from token arithmetic.

6. Exit task: a checkable engineering decision exercise

Pick one task with discrete answers (e.g., a math word-problem subset) and one fixed base model checkpoint. Do not edit chapters 21–22; this lesson stands alone.

1. State the budget. Token budget B, average candidate length L (assumed), latency ceiling, N candidates, parser, and the independent correctness verifier you will use to measure (not r̂).
2. Calculate expected candidates. With an assumed p (state it as an assumption), compute expected correct candidates, P(at least one correct), and P(all wrong) for N = 4, 8, 16.
3. Run and measure. For each N, generate N candidates, apply each selection method, and measure: parse rate, token spend, and wall-clock latency (label every number measured or calculated). Report accuracy differently by method: for best-of-N and self-consistency, report single-answer selection accuracy (independent verifier); for repeated sampling, report P(at least one correct among N) = 1 − (1 − p)ᴺ, since repeated sampling returns all N candidates with no single selection — its accuracy is not comparable on the same axis.
4. Decide. Adopt best-of-N only if measured accuracy exceeds self-consistency at the same N and reward-hacking is not visible (accuracy does not fall as N increases). Adopt self-consistency only if parse rate is high and no single wrong answer dominates. Otherwise reject or stay with repeated sampling / a single greedy decode.
5. Verification gaps. List at least three items you did not measure (e.g., long-tail reward-model error, parse rate on out-of-distribution inputs, latency at batch size > 1, r̂ calibration drift).

The exercise must not claim improvements that are not reproduced. Any number without a measurement source is calculated or illustrative.

Exercise: compare two selection methods under a fixed budget

1. 01With B = 2,000 tokens, L = 250 tokens per candidate, and assumed p = 0.25, compute Nmax, expected correct candidates, P(at least one correct), and P(all wrong) for N = 4 and N = 8. Label every result calculated.

   Show a hint

   Nmax = floor(B / L). P(at least one correct) = 1 - (1 - p)^N.

2. 02Describe one failure case for best-of-N and one for self-consistency in your own words, and state one observable check that would catch each before deployment.

   Show a hint

   For best-of-N: reward hacking — check accuracy vs N with an independent verifier. For self-consistency: shared blind spot — inspect whether losing candidates share the same error.

3. 03Write the budget and decision rule for adopting best-of-N on a task of your choice. State what you would measure, the pass condition, and at least two verification gaps.

   Show a hint

   Pass condition could be: measured self-consistency accuracy at N = 8 exceeded by best-of-N at N = 8, with no accuracy decline from N = 4 to N = 16.

Related lessons

• Lesson 21 (how-reasoning-models-work): reasoning models emit intermediate tokens; this lesson spends inference-time compute on top of a base policy.
• Lesson 22 (evaluating-reasoning): baselines, answer normalization, extraction, and parsing failures — the parser and independent verifier used here.
• Lesson 20 (architecture-comparison): fixed-baseline comparison framework; this lesson applies it to inference-time selection methods.

Related lessons

• LLM Internals & Architecture · Reasoning models and test-time computationHow Reasoning Models Work: Training Paths, Traces, and Cost16 min
• LLM Internals & Architecture · Reasoning models and test-time computationEvaluating Reasoning: Baselines, Verifiers, and Parsing Failures12 min
• LLM Internals & Architecture · Modern LLM architecturesArchitecture Comparison: Documenting a Defensible Choice14 min

Sources and claim-to-source mapping

Source	Exact section	What it supports	Scope and limits
Wang et al. (2022), “Self-Consistency Improves Chain of Thought Reasoning in Language Models” (arXiv:2203.11171)	§2 (method), §3.2 (results), §3.4 (robustness/scaling)	Self-consistency samples diverse reasoning paths and returns the plurality-vote answer; over greedy decode on 137B LaMDA-PT: GSM8K +10.6%, SVAMP +14.4%, MultiArith +23.9%; on code-davinci-001: MultiArith +23.2%. Plurality vote outperformed probability-weighted aggregation; accuracy saturates as N grows large	Gains are specific to the cited models (137B LaMDA-PT; code-davinci-001), benchmarks (GSM8K, SVAMP, MultiArith), and prompts; they do not transfer to your task or model
Huang et al. (2025), “Is Best-of-N the Best of Them?” (arXiv:2503.21878)	§3.1 (BoN guarantee), Abstract	Best-of-N maximizes an imperfect reward model over N candidates; regret bound has an overoptimization term that grows with N; accuracy can degrade as N increases past a threshold (reward hacking)	Theoretical framework with specific coverage assumptions; empirical results on specific tasks and reward models
Cobbe et al. (2021), “Training Verifiers to Solve Math Word Problems” (arXiv:2110.14168)	§4.2	An outcome-based verifier can rerank candidate solutions and improve math accuracy	Verifier is trained separately; specific dataset and model scale

Verification gaps stated:

• All arithmetic in sections 4.1–4.3 is calculated from stated probability models (p, W, B, L); it has not been measured against a real model.
• The table-3 toy selection example is constructed to illustrate mechanisms; it is not a model output.
• Primary-source numbers (GSM8K +10.6%, etc.) are specific to the cited models, datasets, and prompts; they do not transfer to your task.
• Reward-hacking and shared-blind-spot claims are qualitatively supported by the cited papers; their occurrence and magnitude on your task require measurement.