Architecture Comparison: Documenting a Defensible Choice

Choosing an architecture is not a beauty contest between papers. A defensible comparison requires a fixed baseline, explicit conditions, and measured deltas — not parameter counts from a marketing slide. This lesson builds the framework that turns architecture choices into documented engineering decisions.

The one-sentence version

A defensible architecture comparison fixes one baseline model and evaluation set, states the exact quality, memory, compute, and hardware conditions, compares each candidate (dense, GQA, MoE, MLA, multi-token, pruned) against the same conditions, separates estimated arithmetic from measured latency, and adopts a change only when all stated thresholds are met by actual measurements.

Everything else is a mechanism for making that sentence operational.

1. What this framework compares

We treat the decoder-only architecture as a family of design choices that can be changed independently. The framework does not rank them; it provides the structure to compare any subset defensibly.

Design dimension	Options we reference (primary sources)	What changes in the architecture
Attention mechanism	MHA (Vaswani 2017 §3.2.2), MQA, GQA (Ainslie et al. 2023), MLA (DeepSeek-V2 §2)	Projection shapes, KV-cache size, head grouping
MLP / feed-forward	Dense FFN (Vaswani 2017 §3.3), narrowed MLP (He et al. 2024 §3.2), MoE (Shazeer 2017; Lepikhin 2020 §2.1; Fedus 2022 §2.2)	Active parameters, dispatch overhead, communication
Prediction objective	Next-token only, multi-token auxiliary (He et al. 2024; DeepSeek 2024)	Training signal, inference path unchanged
Structural removal	Block drop, MLP drop, attention drop (He et al. 2024 §3.1–3.3)	Depth, capacity, recovery-training requirement

The framework applies to any of these. It does not assume that any option is universally better.

2. The fixed-baseline contract

Every comparison in this lesson uses the same contract. Changing any part of it invalidates cross-variant comparison.

• Baseline model: a single pre-trained decoder-only checkpoint with fixed L, d_model, H_q, d_head, d_ff, vocabulary V, and context window S_max. No two baselines.
• Evaluation set: the same held-out set for all variants. Not the training set. Not a mix of benchmarks chosen per variant.
• Metric: perplexity for language modelling, or a task-specific accuracy (e.g., MMLU subset, HumanEval) for capability evaluation. The same metric for all variants.
• Hardware: a single GPU model (e.g., NVIDIA A100) with fixed driver and CUDA version. No multi-device routing that introduces inter-device overhead unless the comparison explicitly includes that condition.
• Serving framework: a standard dense-serving framework (e.g., vLLM) with stated batching mode (continuous batching disabled for simplicity, or explicitly enabled with the same settings for all variants).
• Batch settings: B (batch size), S (sequence length), and S_prefill (prefill length) stated exactly, identical across variants.

Why the contract matters

Changing the evaluation set between variants makes the quality delta incomparable. Changing the batch size changes the memory-bandwidth bottleneck, which changes which architecture looks faster. The contract is not bureaucracy; it is the condition under which any comparison claim is meaningful.

Comparison framework

From fixed baseline to defensible decision: variants pass through the same measurement gates before adoption.

Read this diagram as text

Baseline model feeds into architecture variants (attention, MLP, sparse, structural). Each variant passes through quality, memory, and latency measurement using the same evaluation set, batch settings, and hardware. Results converge at a decision gate with explicit thresholds. The framework is illustrative; measurement values must come from the user's target setup.

3. Candidate configurations — with exact primary-source dimensions

We describe each candidate using the exact dimensions reported in the primary source. We do not invent new numbers.

3.1 Dense MHA baseline (Vaswani 2017)

From Vaswani et al. (2017), §3.2.2 and §3.3:

• d_model = 512
• h = 8 parallel heads, d_k = d_v = 64
• W_Q, W_K, W_V: 512 × 64; W_O: (8 · 64) × 512
• d_ff = 2048 (4 · d_model)
• Residual connections around both sub-layers; layer normalisation applied before the sub-layer (pre-norm in modern variants, post-norm in the original).

This is the reference architecture. Every comparison starts from a model that uses these exact dimensions, or from a stated modern variant (e.g., d_model = 4096, h = 32) that is explicitly declared as the new baseline.

3.2 Dense GQA: Llama-2 70B (Touvron et al. 2023, Table 1)

From Touvron et al. (2023), Llama 2 paper, Table 1 and model description:

• L = 80 layers
• d_model = 8192
• H_q = 64 query heads, H_kv = 8 key/value heads (GQA applied only to 70B)
• d_head = 128
• d_ff = 28672 (3.5 · d_model)
• Vocabulary V = 32000
• Context window 4096 (doubled from Llama 1)

The GQA change does not alter the attention mechanism’s mathematical form; it changes the projection shapes of W_K and W_V so that the same key/value head serves multiple query heads. The KV-cache size per token is reduced by a factor of H_q / H_kv = 8 relative to MHA with the same d_model and h.

3.3 Dense MoE: Mixtral 8x7B (Jiang et al. 2024, Abstract; §2)

From Jiang et al. (2024), Mixtral of Experts:

• N = 8 experts per MoE layer, k = 2 selected per token (top-2 routing)
• L = 32 layers; the FFN sub-block of every layer is replaced by an MoE layer (8 experts, top-2) — all 32 layers, not a selected subset
• Total stored parameters: 47B (paper: 46.7B)
• Active parameters per token: 13B (paper-reported)
• Each expert is a full FFN the same size as the dense FFN it replaces: d_model = 4096, expert hidden 14336 (3.5 · d_model)

The comparison framework must include the same L, same expert size, and same evaluation set. We compare Mixtral against a dense model with the same d_model and expert MLP size under the same conditions — not against a different dense baseline chosen to make 47B look small.

3.4 DeepSeek-V2: MLA + DeepSeekMoE (DeepSeek-AI 2024, §Abstract; §2; §3)

From DeepSeek-AI (2024), DeepSeek-V2 technical report:

• Total parameters: 236B; active per token: 21B (8.9% of total)
• L = 60 layers, d_model = 5120, n_h = 128 attention heads, d_h = 128
• Multi-head Latent Attention (MLA): compressed KV representation (c_t^KV, d_c = 512) cached; decoupled RoPE key (k_t^R, d_h^R = 64) cached separately; keys and values reconstructed before attention
• MoE routing (DeepSeekMoE): 160 routed experts + 2 shared experts per layer; 6 routed experts active per token (2 shared experts always active, 8 total active experts per token); the first layer keeps a dense FFN
• Context window: 128K tokens

The architecture comparison must separate the attention-side effect (MLA, KV-cache memory) from the MLP-side effect (DeepSeekMoE, active compute). Mixing the two makes attribution impossible — the paper’s headline figures compare the combined MoE + MLA model against DeepSeek 67B (dense MHA), not either change in isolation (see section 6.3).

3.5 Multi-token prediction (Lesson 18; He et al. 2024)

From the multi-token prediction design (see Lesson 18):

• An auxiliary head predicts n future tokens from the same hidden state.
• The inference path remains the standard next-token loop; only the training objective changes.
• The comparison framework treats this as a training-side change, not a serving-side change. If the claim is about inference latency or memory, multi-token prediction should not be included; if the claim is about quality per training compute, it should.

Do not mix training-side and serving-side claims

Multi-token prediction, structural pruning, and MoE routing all change different parts of the lifecycle. A defensible comparison groups variants by the dimension they change: attention (MLA, GQA), MLP (dense, narrow, MoE), structure (pruned), or objective (multi-token). Comparing across groups requires stating the dimension explicitly.

3.6 Structural pruning (Lesson 19; He et al. 2024, §3.1–3.3; Figure 3; Table 2)

From He et al. (2024), “What Matters in Transformers? Not All Attention is Needed”:

• Three removal forms: Block Drop (§3.1), MLP Drop (§3.2), Attention Drop (§3.3).
• Similarity-based redundancy metric: cosine similarity between input and output hidden states of a layer; layers with high similarity are selected for removal.
• Quality results in Table 2 are reported without full recovery training. Any adoption claim requires recovery-training measurement.
• The similarity metric measures redundancy, not functional importance (§4.3).

The comparison framework treats structural removal as a separate architecture family: you compare the original dense model to the pruned model with the same L', d_model, and d_ff', measured on the same evaluation set, with recovery training applied or explicitly stated as not applied.

4. Explicit conditions: the measurement table

Every adoption claim must reference this table. Each cell is either a measured value (from your run) or a stated condition (from your contract). Empty cells mean the comparison is incomplete.

Condition category	Required statement	Example (stated condition, not measured)	Why it must be stated
Quality metric	Perplexity or task accuracy, same metric for all variants	Perplexity on a 10K-token held-out set from domain X	A different metric hides the tradeoff
Quality threshold	Maximum acceptable degradation (e.g., perplexity increase ≤ 3%)	Perplexity increase ≤ 2% absolute	Without a threshold, any result looks acceptable
Memory budget	Peak GPU memory at B, S_max, stated precision	Peak memory ≤ 36 GB at B = 8, S_max = 2048, FP16	Without budget, memory reduction is meaningless
Compute budget	FLOPs per token or total training compute, stated exactly	Compute per token measured by framework profiler	FLOP estimates are not latency
Hardware	GPU model, driver, CUDA version, single-device or multi-device	NVIDIA A100 80 GB, CUDA 12.2, single device	Different hardware changes the bottleneck
Batch settings	B, S_prefill, S_max, continuous batching enabled/disabled	B = 4, S_prefill = 512, S_max = 1024, continuous batching disabled	Batch changes the memory-bandwidth vs compute balance
Serving framework	Framework name and version, fusion settings	vLLM v0.6, default fusion, no speculative decoding	Framework overhead is not architecture overhead
Recovery cost	Whether recovery training is applied, dataset, steps, learning rate	Recovery: 5% of pretraining data, 10K steps, LR 1e-5	Without this, structural removal quality is incomparable

A comparison that does not state all nine conditions is incomplete. A comparison that states them but does not measure the outcome is a proposal, not a result.

5. Original ablation plan: cross-architecture comparison

This ablation plan extends Lesson 19’s structural-removal framework to cross-architecture decisions. It is a design proposal, not an executed experiment. Any adoption requires measurement on the user’s target setup.

5.1 Fixed baseline (same contract)

• Decoder-only, L = 32, d_model = 4096, d_ff = 16384 (4 · d_model), V = 32000, context S_max = 4096.
• Checkpoint: a pre-trained dense MHA model with known perplexity on the target evaluation set.
• No structural changes applied to the baseline.

5.2 Variants to compare

Variant	Architecture change	Source reference
A (attention)	Convert MHA to GQA (H_q = 32, H_kv = 8, same d_model)	Ainslie et al. (2023), GQA paper
B (attention, compression)	Replace MHA with MLA (d_c compressed latent, d_h^R decoupled RoPE)	DeepSeek-AI (2024), DeepSeek-V2 §2–§3
C (MLP, sparse)	Replace MLP with MoE (N = 8, k = 2, expert size = original d_ff)	Jiang et al. (2024), Mixtral §Abstract
D (MLP, narrow)	Reduce d_ff by 25% (16384 → 12288)	He et al. (2024), MLP Drop §3.2
E (structure)	Remove 25% of attention sub-layers using similarity metric	He et al. (2024), Attention Drop §3.3

5.3 Metrics and thresholds

• Quality: perplexity on held-out set; threshold: increase ≤ 3% relative to baseline.
• Memory: peak GPU memory at B = 4, S_max = 2048, FP16; threshold: reduction ≥ 15% relative to baseline.
• Latency: decode inter-token latency (ITL) measured with framework profiler; threshold: reduction ≥ 10% at B = 4, S_max = 1024.
• Prefill latency (TTFT): measured at B = 4, S_max = 1024; reported alongside ITL. Any increase versus baseline must be documented and accepted before adoption.
• Recovery cost: for variants A, B, D, E, document whether recovery training is applied; for variant C (MoE), document whether expert weights are initialised from dense weights (optional initialisation choice, not a paper claim) and whether continued training is applied.

5.4 Decision rule (same as Lesson 19, extended)

A variant is adopted only if all of the following are met by measured values, not estimates:

1. Quality delta is within the stated threshold.
2. Memory reduction is within the stated budget.
3. Latency reduction is within the stated budget (ITL reduction ≥ 10% at B = 4, S_max = 1024), and TTFT is measured; any TTFT increase versus baseline is documented and accepted.
4. Recovery-training cost is documented and accepted.
5. The dimension being changed (attention, MLP, structure, objective) is stated explicitly, and no cross-dimension claim (e.g., “MoE improves attention”) is made.

If any condition fails, the variant is rejected or returned for further tuning.

A common comparison mistake

Comparing parameter counts across architecture families (e.g., Mixtral 47B total vs Llama-2 70B) without fixing the evaluation set, batch settings, and hardware produces numbers that look comparable but are engineering noise. The framework prevents this by making the contract explicit before any measurement begins.

6. Illustrative arithmetic: labelled estimates only

The numbers below are calculated from stated dimensions. They are illustrative, not measured. They show how to separate estimated savings from what must be measured.

Assume a toy dense MHA baseline (d_model = 768, L = 12, d_ff = 3072, V = 50257, B = 1, S_max = 1024, b = 2 bytes FP16):

6.1 Attention-side estimates (primary-source dimensions)

Architecture	Attention params per layer	Total attention params (L = 12)	KV-cache bytes per layer at S_max = 1024, B = 1, FP16	Source reference
MHA (H_q = H_kv = 12, d_head = 64)	4 · 768² = 2,359,296	28,311,552	2 · 12 · 1024 · 64 · 2 = 3,145,728 (3.00 MiB)	Vaswani 2017 §3.2.2
GQA (H_q = 12, H_kv = 3, same d_head)	W_Q: 768 × (12·64) = 589,824; W_K, W_V: 768 × (3·64) = 147,456 each; W_O: (12·64) × 768 = 589,824; total 1,474,560	17,694,720	2 · 3 · 1024 · 64 · 2 = 786,432 (0.75 MiB)	Ainslie et al. 2023
GQA reduction vs MHA	37.5%	37.5%	75% (from 12 to 3 KV heads)	Calculated

These are parameter-count and memory-formula estimates. Actual latency requires framework profiling; GQA reduces KV-cache memory but does not guarantee faster decode if the bottleneck is weight loading, not cache size.

6.2 MLP-side estimates

Variant	MLP params per layer (W_1, W_2)	Total MLP params (L = 12)	Active params per token (MoE only)	Source reference
Dense (d_ff = 3072)	2 · 768 · 3072 = 4,718,592	56,623,104	Full 4.72M	Vaswani 2017 §3.3
Narrowed (d_ff = 2304, −25%)	2 · 768 · 2304 = 3,538,944	42,467,328	Full 3.54M	He et al. 2024 §3.2
MoE (N = 8, k = 2, expert size = d_ff)	8 · 4,718,592 = 37,748,736 stored	452,984,832 stored	2 · 4,718,592 = 9,437,184 (2 experts active)	Jiang et al. 2024; Shazeer 2017

The MoE variant stores 8× the MLP parameters and activates 2/8 = 25% of the stored MLP parameters per token — but against this dense baseline, active per-token MLP compute is 2 × 4.72M = 9.44M, roughly 2× the dense baseline’s per-token MLP (4.72M). The 25% figure is a share of stored parameters, not a saving versus the dense baseline: MoE buys 8× stored capacity at about 2× the per-token MLP compute of an equally wide dense model. The active-compute saving only appears against a dense model with the same total stored size. The framework must measure whether that tradeoff lowers measured latency at the target batch size; it often does not if the bottleneck is memory bandwidth for loading all N expert weights, or if inter-device routing dominates.

6.3 DeepSeek-V2: combined attention + MoE estimate

Dimensions from DeepSeek-AI (2024), §Abstract and §3.1.2: L = 60, d_model = 5120, n_h = 128, d_h = 128, d_c = 512 (4 · d_h), d_h^R = 64 (d_h / 2); MoE: 160 routed experts + 2 shared, 6 routed active per token; 236B total / 21B active (8.9%).

Per Table 1 of the paper, MLA’s structural KV cache is (d_c + d_h^R) · l ≈ 4.5 · d_h · l elements per token per layer versus MHA’s 2 · n_h · d_h · l — model-specific, not a universal ratio. The headline 93.3% KV-cache reduction and 5.76× throughput (Abstract; §3.2.3) compare DeepSeek-V2 (MoE + MLA, with KV-cache quantized to roughly 6 bits per element in deployment) against DeepSeek 67B (dense MHA). Neither figure isolates MLA from MoE, and both depend on the paper’s exact setup; do not attribute either to one change alone.

The framework treats the two dimensions separately: MLA changes KV-cache memory and attention-side compute; DeepSeekMoE changes MLP compute and stored memory. A comparison that attributes a latency improvement to MoE when the bottleneck is KV-cache size is misattributing the mechanism.

Label every arithmetic claim

Any number in this lesson without an explicit measurement source must be labelled “calculated from stated dimensions” or “reported by the primary source for the specific model and setup.” The framework requires this label; the comparison report (exit task) must apply it to every number presented.

7. Sources and verification gaps

Source	Exact section / table	What it supports in this framework	Scope and limits
Vaswani et al. (2017), “Attention Is All You Need” (arXiv:1706.03762)	§3.2.2 (multi-head attention definition, Concat + W^O); §3.3 (position-wise FFN, d_ff = 4 · d_model)	MHA projection shapes; MLP expansion factor; original decoder architecture reference	Original encoder-decoder model (d_model = 512, h = 8); does not address modern GQA, MoE, or MLA
Ainslie et al. (2023), GQA paper	§4.3 (uptraining MHA to GQA); model card (Llama-2 70B GQA)	GQA projection shape change; KV-cache reduction; uptraining recovery	Results reported for Llama-2 70B conversion; not universal
Touvron et al. (2023), Llama 2 (arXiv:2307.09288)	Table 1 (model dimensions); §2 (GQA in 70B); §3.1 (pretraining settings)	Exact L, d_model, H_q, H_kv, d_ff, V, context window for Llama-2 family	Results for Llama-2 models only; not transferable without measurement
Jiang et al. (2024), Mixtral 8x7B (arXiv:2401.04088)	Abstract (N = 8, k = 2, 47B total / 13B active); §2 (MoE layer placement; FFN sub-block replaced in all 32 layers)	Active vs stored parameter distinction; top-2 routing; every layer’s FFN sub-block is MoE	Results for Mixtral 8x7B; not universal to all MoE configurations
Lepikhin et al. (2020), GShard (arXiv:2006.16668)	§2.1 (MoE layer placement); §2.2 (capacity factor C, group capacity)	Capacity overflow definition; expert dispatch mechanism	Translation task results; not universal benchmarks
Fedus, Zoph & Shazeer (2022), Switch Transformer (arXiv:2101.03961)	§2.2 (auxiliary loss alpha = 1e-2); §2.3 (capacity, drop rates < 1%); §3 (C4 masked LM results)	Simplified load-balancing; capacity handling; drop-rate thresholds	C4 masked-language-modelling results; not universal quality claims
Shazeer et al. (2017), Sparsely-Gated MoE (arXiv:1701.06538)	§2.1 (softmax gating); §2.3 (load-balancing and importance-weighting losses)	Original MoE routing design; top-k selection	Language modelling and translation results up to 137B parameters
DeepSeek-AI (2024), DeepSeek-V2 (arXiv:2405.04434)	Abstract (236B total / 21B active, 128K context); §2 (MLA design, c_t^KV, k_t^R); §3.1.2 (model hyper-parameters); §3.2.3 (efficiency)	MLA compression mechanism; DeepSeekMoE: 160 routed + 2 shared experts, 6 routed active per token; KV-cache reduction (93.3%) and throughput (5.76×) reported vs DeepSeek 67B, not MLA-vs-MHA	DeepSeek-V2 results; MLA reduction is model-specific; headline figures combine MoE + MLA + deployment quantization
He et al. (2024), “Not All Attention is Needed” (arXiv:2406.15786)	§3.1 (Block Drop); §3.2 (MLP Drop); §3.3 (Attention Drop); Figure 3 (removal forms); Table 2 (quality results); §4.3 (recovery training not measured)	Three structural removal forms; similarity metric; attention removal retains quality best; no full recovery-training measurement	Measured on authors’ model family; results without recovery training

Verification gaps stated:

• The parameter and memory calculations in section 6 are calculated from stated dimensions; they have not been executed against a real checkpoint.
• The latency effects discussed in sections 3, 5, and 6 are discussed from first principles and primary-source reports; any specific latency claim requires measurement on the user’s target framework, GPU, batch size, and sequence length.
• The comparison framework (section 5) is a design proposal, not a completed experiment. Any adoption decision requires the measured values described in the decision rule.
• Multi-token prediction (section 3.5) is treated as a training-side change; no inference-side latency or memory claim is made for it.
• DeepSeek-V2’s KV-cache reduction (93.3%) and throughput (5.76×) are reported for DeepSeek-V2 vs DeepSeek 67B (MoE + MLA vs dense MHA); they are not MLA-vs-MHA ratios and do not transfer to other MLA implementations or setups.

8. Exit task: a defensible comparison report

Write a short report (maximum 600 words) comparing one architecture variant to the fixed baseline from section 5.1. The report must include:

1. Baseline and variant: state the model dimensions (L, d_model, d_ff, H_q, d_head, V, S_max), the structural or architecture change applied, and the primary source that defines it (with exact section).
2. Measured results (or stated as not measured): for the variant, report measured perplexity (or accuracy), measured peak memory, measured TTFT and ITL at B = 4, S_max = 1024 (or your chosen settings). If you have not executed the measurement, state clearly: “Not measured; the numbers below are estimated only.”
3. Estimated changes: calculate parameter reduction and memory reduction using the formulas from section 6. Label every arithmetic result as “estimated from stated dimensions, not measured.”
4. Decision: apply the decision rule from section 5.4. State adopt, reject, or conditional, with the exact conditions (e.g., “adopt GQA only if KV-cache memory is reduced by ≥ 50% at B = 8, S_max = 2048, and perplexity increase ≤ 2%, with recovery training applied on 5% of pretraining data”).
5. Verification gaps: list at least three unverified items (e.g., multi-batch latency, long-context behaviour beyond S_max, out-of-distribution input quality, recovery-training cost not measured, framework overhead not separated from architecture overhead).

The report must not claim benchmark improvements that have not been reproduced. Any number without a measurement source must be labelled estimated or illustrative.

Exercise: Defensible architecture comparison report

1. 01Using the framework from section 2, select one variant from section 5.2 (A through E). Write a 300-word report comparing it to the fixed baseline in section 5.1. State the exact dimensions, the change made, the measurement conditions, and the decision rule applied.

   Show a hint

   Keep the baseline fixed. Reference the exact primary-source section (e.g., Ainslie 2023 §4.3 for GQA). Label every arithmetic claim.

2. 02Calculate the estimated parameter reduction for your selected variant using the formulas in section 6. State the exact arithmetic, the dimensions used, and label the result clearly.

   Show a hint

   Use the per-layer formulas: MHA attention ≈ 4 · d_model²; MLP ≈ 2 · d_model · d_ff. Scale by L. Label every number.

3. 03Identify at least three verification gaps in your proposed comparison. For each gap, state what measurement would close it and why the comparison is incomplete without it.

   Show a hint

   Consider batch-size variation, framework overhead, recovery-training cost, out-of-distribution quality, and long-context behaviour.

9. Related lessons

• Lesson 15 (mha-mqa-gqa): attention architecture tradeoffs; the GQA and MLP dimension mechanics that any comparison of attention-side changes must preserve.
• Lesson 16 (mixture-of-experts): MoE routing, capacity, and load balancing; the active-vs-stored parameter distinction that comparison reports must keep separate.
• Lesson 17 (multi-head-latent-attention): MLA compression, RoPE decoupling, and the KV-cache reduction mechanism; the attention-side comparison for DeepSeek-style architectures.
• Lesson 18 (multi-token-prediction): auxiliary multi-token objectives; the training-side comparison that must not be confused with serving-side latency claims.
• Lesson 19 (structural-pruning-transformers): the structural-removal ablation framework that feeds directly into this lesson’s cross-architecture comparison design.

Related lessons

• LLM Internals & Architecture · Modern LLM architecturesMHA vs. MQA vs. GQA: Attention Architecture Tradeoffs14 min
• LLM Internals & Architecture · Modern LLM architecturesMixture of Experts: Routing, Dispatch, and Load Balancing15 min
• LLM Internals & Architecture · Modern LLM architecturesMulti-Head Latent Attention: DeepSeek's KV-Cache Design14 min
• LLM Internals & Architecture · Modern LLM architecturesMulti-Token Prediction: Objectives and Tradeoffs15 min
• LLM Internals & Architecture · Modern LLM architecturesWhat Can You Remove From a Transformer?14 min

Interview questions

Why does a defensible architecture comparison require a fixed baseline and the same evaluation set for every variant?

Changing the baseline or evaluation set between variants makes the quality delta incomparable. The framework requires a single checkpoint, a single held-out set, and identical batch/hardware settings so that any measured difference can be attributed to the architecture change, not to measurement noise.

What is the difference between estimated parameter reduction and measured latency reduction, and why does the framework separate them?

Estimated parameter reduction is calculated from architecture dimensions (e.g., MHA to GQA reduces attention params by 37.5%). Measured latency reduction depends on the target framework, GPU, batch size, sequence length, and whether the bottleneck is memory bandwidth or compute. The framework requires both to be stated separately; only measured values justify adoption.

When comparing a MoE variant to a dense baseline, what two attribution errors must the comparison avoid?

First, attributing a latency improvement to MoE when the bottleneck is actually attention-side (e.g., KV-cache size from MHA vs GQA). Second, comparing total stored parameters (47B for Mixtral) to dense parameters (70B for Llama-2) without fixing the evaluation set, batch settings, and hardware. The framework prevents both by requiring explicit conditions and separate attribution for attention vs MLP changes.