Evaluating Reasoning: Baselines, Verifiers, and Parsing Failures

Reasoning models emit intermediate tokens before a final answer. Lesson 21 showed that a correct answer does not prove the trace is faithful. This lesson asks a different question: how do you know the evaluation itself is trustworthy? A parser that drops unparseable answers, a normalizer that strips units too aggressively, or a verifier that agrees with the model’s blind spots can all create the appearance of improvement where none exists.

The one-sentence version: Establish a fixed baseline on a held-out dataset, extract and normalize answers carefully, count parser failures separately, apply verifiers knowing their limits, compute exact match and pass@k, and classify every error before claiming a model got better.

---

1. Establish a baseline first

Before changing anything — prompt, temperature, model, extractor, or verifier — run one fixed configuration on a fixed, held-out dataset and record the result. That is your baseline. Every later change is compared against it, not against a vague memory of “it seemed better.”

A baseline needs:

• A fixed dataset (problems + ground-truth answers), held out from any training or prompt design.
• Fixed prompts and few-shot exemplars, if any.
• Fixed temperature and random seeds.
• A fixed answer extractor and normalizer.

Report exact match (EM): the fraction of problems where the extracted, normalized answer equals the ground truth. Report it with a confidence interval; a small evaluation set has a wide interval, and a one-point swing on twenty problems is noise.

Source-bound claim: GSM8K uses exact numeric match as its correctness criterion (Cobbe et al., 2021). Answer extraction materially impacts reported accuracy: a correct calculation rendered as “18 dollars” scores zero if the evaluator expects “18” unless normalization strips units and currency.

---

2. Build a small dataset (illustrative)

Do not train on the test set. For a reproducible teaching example, hand-write a tiny set of problems covering distinct failure modes:

#	Problem	Ground truth	Intended trap
1	A tank holds 500 liters. It is 40% full. 30 liters are added. How many liters are in the tank?	230	percentage computed on the wrong base
2	A hall has 12 rows of 4 chairs. 7 chairs are removed. How many remain?	41	invented premise compensates a wrong product
3	A shirt costs $20, then 25% off, then 10% tax. Final price?	16.50	tax applied to original instead of discounted price
4	Convert 3 kilometers to meters.	3000	unit confusion
5	What is 7 + 8 × 2?	23	operator precedence
6	A car travels 60 km/h for 2.5 hours. Distance?	150	unit/decimal handling

Label as original: This six-problem set is an original illustrative example, not a published benchmark. Its difficulty and discrimination are not validated. Use it to practice the evaluation pipeline; do not cite its results as evidence about any model’s capability.

---

3. Answer extraction and normalization

The model output is free text. Extract a candidate answer, then normalize before comparing:

1. If a \boxed{} delimiter is present, take its contents (EvalScope/GSM8K convention: “put your final answer within \boxed”).
2. Otherwise, take the last number-like token in the output.
3. Normalize: remove currency symbols and units, replace commas (“1,024” → “1024”), lowercase, trim whitespace, canonicalize decimals.

Danger: overly aggressive normalization creates false positives. If a problem asks for cents and the model answers in dollars, stripping units makes an incorrect answer appear correct. Normalize, but record the raw extraction next to the normalized value so you can audit false positives.

Executable extractor (deterministic, no model needed)

import re

def extract_answer(text: str) -> str | None:
    """Return the raw candidate answer, or None if unparseable."""
    m = re.search(r"\\boxed\{([^}]*)\}", text)
    if m:
        return m.group(1).strip()
    # Fallback: last standalone number-like token (non-capturing group so
    # findall returns the full match, not just the decimal part).
    tokens = re.findall(r"-?\d[\d,]*(?:\.\d+)?", text)
    return tokens[-1] if tokens else None


def normalize(answer: str) -> str:
    """Strip units/currency/commas; canonicalize decimals; lowercase; trim."""
    a = answer.lower().strip()
    a = re.sub(r"[^\d.,\-]", "", a)  # keep digits, dot, comma, minus
    a = a.replace(",", "")
    # Canonicalize decimals: strip trailing zeros after the decimal point
    # so 16.50 and 16.5 compare equal.
    if "." in a:
        a = a.rstrip("0").rstrip(".")
    return a


def correct(raw: str, ground_truth: str) -> bool | None:
    """Return True if correct, False if wrong, None if a parser failure."""
    ex = extract_answer(raw)
    if ex is None:
        return None  # parser failure, not correct and not wrong
    return normalize(ex) == normalize(ground_truth)

Run this on known outputs to confirm behavior. Every row was executed against the code above; the results below are actual outputs, not illustrations.

Raw model output	extract_answer	normalize	ground truth (normed)	result
\boxed{230}	230	230	230	CORRECT
The answer is 16.50 dollars.	16.50	16.5	16.5	CORRECT
I think it's about forty.	None	—	40	PARSER FAILURE
1,024 meters	1,024	1024	1024	CORRECT
42	42	42	42	CORRECT (integer)
3.14	3.14	3.14	3.14	CORRECT (decimal)
1,000 kg	1,000	1000	1000	CORRECT (commas)
\boxed{7}	7	7	7	CORRECT (boxed)
The answer is approx forty.	None	—	40	PARSER FAILURE
Result: -5	-5	-5	-5	CORRECT (signed)

Across these 10 executed rows: 8 correct, 0 wrong, 2 parser failures.

• Overall exact match rate: 8/10 = 80%
• Parser-failure rate: 2/10 = 20%

Report both together: exact match alone (80%) hides that 20% of outputs were never answers at all. The two failures are format problems, not reasoning problems.

---

4. Count parser failures separately

An output from which no candidate answer can be extracted is a parser failure — neither correct nor incorrect. Count it as its own bucket:

• Do not silently drop parser failures.
• Do not count them as correct.
• Do not count them as wrong answers (they are not answers at all).

Report the parser-failure rate alongside exact match. A model with 70% exact match and 20% parser failures is different from a model with 70% exact match and 2% parser failures.

---

5. Verifiers and their limits

A verifier scores candidate solutions (process reward model or outcome verifier). Know what it can and cannot do:

• Process supervision scores intermediate steps (Lightman et al., 2023; Uesato et al., 2022). It is expensive, and its reward model is a proxy that can be wrong or gamed. It is not a proof.
• Outcome-based verifiers rerank candidates by final answer (Cobbe et al., 2021). The signal is outcome-based, used for selection.
• Verifier blind spots: a verifier can miss unsupported premises, reward-hack toward surface features, and share the model’s blind spots. If N samples all look correct to the verifier, that does not validate those N samples — it only confirms they look plausible to the verifier.

Source-bound claim: Cobbe et al. (2021) trained an outcome-based verifier to rerank candidate solutions on GSM8K. Lightman et al. (2023) showed process supervision reduced reasoning errors among final-answer-correct solutions on MATH, using PRM800K with 800,000 step-level labels; Uesato et al. (2022) found outcome supervision produced similar final-answer error with less supervision, but process supervision cut reasoning errors among correct answers (14.0% → 3.4%).

---

6. Metrics

Metric	Definition	Notes
Exact match (EM)	# correct / # total after normalization	Binary, strict; GSM8K convention.
Parser-failure rate	# unparseable / # total	Report separately; do not fold into wrong answers.
pass@1	EM with one sample per problem	Single-sample correctness.
pass@k	E[1 − binom(n−c, k) / binom(n, k)] where n = samples per problem, c = correct (Chen et al., 2021)	Probability at least one of k samples is correct. Requires n ≥ k samples per problem.
Confidence interval	Wilson or bootstrap interval on EM	Report with EM; small sets have wide intervals.

Source-bound claim: pass@k is the unbiased estimator from Chen et al. (2021, arXiv:2107.03374), §2.1. It estimates the probability that at least one of k samples is correct given n samples with c correct.

Example (illustrative arithmetic, not a benchmark): For one problem you sample n = 5 outputs and c = 2 are correct. pass@2 = 1 − binom(3,2)/binom(5,2) = 1 − 3/10 = 0.70: a 70% chance that at least one of two samples is correct.

---

7. Error analysis

For every wrong or unparseable answer, classify into one bucket:

Bucket	Meaning
Parser failure	No extractable answer (format problem).
Normalization failure	Answer extracted correctly but normalization mishandled units/decimals.
Arithmetic error	Correct premise, wrong computation.
Premise error	Used a fact not in the problem, or dropped a real constraint.
Reasoning / logic error	Valid premises, invalid inference.
Final-answer mismatch	Correct trace, but the extracted final answer differs from the trace result.

Report counts per bucket and the denominator. State the confidence interval on EM. Do not claim a universal “reasoning gap” from one small set — report what was observed for this model on these prompts.

---

8. Exit task: run a mini evaluation

Use the six-problem dataset in §2 with the extractor in §3. You do not need a model: supply six hand-written model outputs (one per problem), two correct, two wrong, one parser failure, one normalization trap.

Measurable exit task: Produce a one-page evaluation report containing:

1. The baseline exact match (with confidence interval) and parser-failure rate.
2. The raw extraction and normalized value for each of the six outputs.
3. The error bucket for each non-correct output.
4. A one-sentence judgment: “Did the model improve over baseline, or did the pipeline change?” — supported by the numbers, not by vibes.

Acceptance criteria: the report separates parser failures from wrong answers, states the denominator and an interval, names at least one verifier limit, and does not claim a universal result from six problems.

---

9. Key takeaways

Key takeaways

• Establish a fixed baseline on a held-out dataset before changing any prompt, model, or extractor.
• Exact match after normalization is the GSM8K convention; extraction and normalization change the number you report.
• Parser failures are not wrong answers — count them as their own bucket and report the rate.
• Verifiers are proxies with blind spots; verifier agreement does not validate the samples it scores.
• pass@k estimates the chance that at least one of k samples is correct; it requires n ≥ k samples per problem.
• Classify every error into parser, normalization, arithmetic, premise, or reasoning buckets before concluding anything.

---

10. Sources and claim-to-source mapping

Source	Direct link	Exact section	What it supports	Scope and limits
Chen et al. (2021), “Evaluating Large Language Models Trained on Code”	https://arxiv.org/abs/2107.03374	§2.1	pass@k formula; functional correctness via unit tests	Code domain; n ≥ k samples required
Cobbe et al. (2021), “Training Verifiers to Solve Math Word Problems”	https://arxiv.org/abs/2110.14168	§4.2	Outcome-based verifier reranks candidates; GSM8K exact-match framing	GSM8K scale; verification ≠ policy update
Lightman et al. (2023), “Let’s Verify Step by Step”	https://arxiv.org/abs/2305.20050	§2.6, §4.1	Process supervision; PRM800K; reduces reasoning errors among correct answers	MATH; reward model is a proxy
Uesato et al. (2022), “Solving math word problems with process- and outcome-based feedback”	https://arxiv.org/abs/2211.14275	§3	Outcome vs process supervision tradeoffs; 14.0% → 3.4% reasoning errors	GSM8K; specific setup
GSM8K dataset (Cobbe et al., 2021)	https://github.com/openai/grade-school-math	README	#### answer delimiter; exact numeric match	Grade-school math; contamination risk
EvalScope GSM8K spec	https://evalscope.readthedocs.io/en/v1.6.1/benchmarks/gsm8k.html	Prompt template	\boxed{} delimiter; numeric extraction	Tool-specific convention
GSM-Symbolic (Apple)	https://machinelearning.apple.com/research/gsm-symbolic	—	GSM8K performance varies with template/numbers changes	Template-variant study

Verification gaps stated:

• The six-problem dataset and worked example are original, illustrative, and not validated; do not cite their results as benchmarks.
• pass@k requires n ≥ k independent samples per problem; miscomputing with n < k is a common error.
• Verifier limits are documented qualitatively; no verifier is validated here.
• Confidence intervals are recommended but not computed in the illustrative arithmetic.

Related lessons

• LLM Internals & Architecture · Reasoning models and test-time computationHow Reasoning Models Work: Training Paths, Traces, and Cost16 min
• LLM Internals & Architecture · Transformer architecture and model inputsPreparing Training Sequences: Shift, Pad, and Pack15 min
• LLM Internals & Architecture · Generation, KV cache, and contextPrefill and Decode: Measuring the Two Phases11 min

---