Diagnose the Gap Before Changing Weights

The one-sentence version

Before fine-tuning any model, evaluate the capability gap across syntax, factual, reasoning, and safety categories on a frozen held-out task set; if prompt engineering, schema-constrained decoding, or retrieval satisfies your operational threshold, or if request volume cannot amortize dedicated GPU hosting, the correct engineering decision is DO NOT TRAIN.

Everything else in this module operationalizes that discipline.

---

Start with the failure: a concrete baseline test

You have an internal domain task—say, diagnosing a database incident from raw telemetry, or calling an internal tool with a strict JSON schema. You grab an off-the-shelf base model (Llama 3 8B Instruct) and run a quick test.

The model fails. It might:

• Hallucinate a command that doesn’t exist in your runbooks (redis-cli --fix-corruption instead of the real redis-cli FAILOVER).
• Output messy markdown (Here's the JSON: followed by a code fence) when your downstream system demands a bare JSON object matching a strict schema.
• Miss proprietary context—it doesn’t know your internal service names, your naming conventions, or that payment-api latency spikes usually mean a Redis primary failure, not a database connection pool issue.

This is the baseline failure. The model has general knowledge, but it doesn’t have your knowledge, your format requirements, or your operational patterns.

---

What is fine-tuning, in plain language?

Fine-tuning means continuing the training process on a smaller, curated dataset of task examples to update the model’s internal weights.

The base model was trained on internet-scale text to predict the next token. Fine-tuning shows it thousands of your input–output pairs (incident → structured triage record) and adjusts the weights so the model internalizes your format, your terminology, and your reasoning patterns.

This contrasts with two lighter-weight alternatives:

Approach	Mechanism	What Changes
Prompt Engineering	Provide instructions, few-shot examples, and schema definitions in the context window at inference time.	Nothing. The model’s weights stay frozen; you steer behavior through the prompt.
Retrieval-Augmented Generation (RAG)	Dynamically retrieve relevant documents (runbooks, API specs, schemas) and inject them into the context window at inference time.	Nothing. The model’s weights stay frozen; you supply up-to-date facts at query time.
Fine-Tuning	Continue training on a curated dataset of (input, desired output) pairs, updating model weights via gradient descent.	The model’s weights change. The behavior becomes baked into the model itself.

---

The four fundamental engineering choices

When your baseline test fails, you have four options—not just “fine-tune or don’t”:

1. Prompting: Can a better prompt fix it?

If the failure is syntax, formatting, or enum confusion (Category A), a tighter system prompt, 3–5 few-shot exemplars, or grammar-constrained decoding (Outlines, vLLM guided_json) often solves it instantly. No weight changes, no hosting cost.

2. Retrieval (RAG): Does the model lack dynamic knowledge?

If the failure is missing facts that change frequently—live database schemas, current inventory, today’s on-call rotation (Category B)—put those facts in a retrieval index. RAG lets you update knowledge by editing documents, not retraining weights.

3. Fine-Tuning: Does the model need to learn a consistent behavior?

If the failure is complex tone, multi-step reasoning patterns, or domain terminology that cannot fit in a prompt (Category C), and you have enough curated examples (≥250 verified pairs) and stable domain definitions, fine-tuning internalizes the pattern so every inference benefits without a long prompt.

4. Holding (Do Not Train): Is the juice worth the squeeze?

If query volume is too low to justify dedicated GPU hosting ($1,350+/month per replica), or prompt engineering is already “good enough” to meet your SLOs, the correct engineering decision is DO NOT TRAIN. Fine-tuning is an architectural commitment with ongoing operational cost—not a free upgrade.

---

The four decision gates

Only after establishing this practical dilemma do we formalize the decision process. Before committing to dataset curation or training in Module 2, the team must pass through four sequential decision gates:

1. Gate 1: Format & Quality — Can prompting, schema-constrained decoding, or RAG meet your operational SLOs (schema 100%, accuracy ≥75%, safety recall ≥95%)?
2. Gate 2: Knowledge & Frontier Models — Does a larger generalist model (e.g., 70B open or commercial API) solve the reasoning gap within your budget?
3. Gate 3: Data Stability — Do you have ≥250 audited training pairs, and does the domain change slower than 20% per quarter?
4. Gate 4: Economics — Does monthly query volume exceed the token amortization break-even (V*), or do policy/air-gapped security constraints prohibit external APIs?

Only when all four gates evaluate to proceed is model adaptation justified.

---

Opening trace: 12-case incident triage test

To ground the decision in measurable reality, consider an autonomous Site Reliability Engineering (SRE) alert triage agent. The agent receives raw production alerts and telemetry, emitting a structured JSON record (IncidentTriageRecord) specifying the primary subsystem, severity, immediate mitigation command, human escalation requirement, and operational reasoning.

Below is the evaluation report from labs/track3/01-baseline-decision/evaluate.py against the 10-record synthetic fixture sample_predictions_zero_shot.json (bracketed [FAIL] and [PASS] annotations added for clarity):

================================================================================
 TRACK 3 MODULE 1: BASELINE DECISION & EVALUATION REPORT
================================================================================
Total Held-Out Tasks:     12
Evaluated Predictions:    10
Missing Predictions:      2

[!] WARNING: MISSING PREDICTIONS DETECTED:
    - Task ID: INC-2026-004 (Scored as 0.0 for accuracy; MISSING (coverage failure, no escalation required))
    - Task ID: INC-2026-012 (Scored as 0.0 for accuracy; MISSING (safety false negative))

--------------------------------------------------------------------------------
 OVERALL TASK QUALITY & SAFETY METRICS
--------------------------------------------------------------------------------
Coverage Rate:               83.3%  (Target: 100.0%)  [FAIL]
Schema Validity Rate:       100.0%  (Target: 100.0%)  [PASS on evaluated]
Subsystem Accuracy:          66.7%  (Target: >= 85.0%) [FAIL]
Severity Accuracy:           66.7%  (Target: >= 85.0%) [FAIL]
Mitigation Accuracy:         33.3%  (Target: >= 80.0%) [FAIL]
Joint Task Accuracy:         33.3%  (Target: >= 75.0%) [FAIL]
Escalation Safety Recall:     0.0%  (CRITICAL Target: >= 95.0%) [CRITICAL FAIL]
Escalation Precision:      N/A (0 predicted positives)  (Target: >= 70.0%)

--------------------------------------------------------------------------------
 PER-SLICE QUALITY BREAKDOWN
--------------------------------------------------------------------------------
Evaluation Slice       | Tasks  | Schema  | Subsys  | Sever   | Mitig   | Joint   | Esc Recall
----------------------------------------------------------------------------------------------
standard_patterns      | 3/4    | 100.0%  |  75.0%  |  75.0%  |  75.0%  |  75.0%  |    100.0%
domain_heuristics      | 4/4    | 100.0%  |  50.0%  |  50.0%  |  25.0%  |  25.0%  |    100.0%
edge_and_safety        | 3/4    | 100.0%  |  75.0%  |  75.0%  |   0.0%  |   0.0%  |      0.0%

The zero-shot fixture fails across multiple dimensions: it drops 2 of 12 cases (83.3% coverage — omitting INC-2026-004, which requires no escalation, and INC-2026-012, which is a safety-critical escalation false negative), misses domain mitigations (33.3%), and exhibits an unacceptable 0.0% escalation safety recall on edge cases (0/4 predicted positives, rendering precision N/A). Note that all 10 present records in sample_predictions_zero_shot.json are syntactically valid JSON (10/10 = 100.0% schema validity on evaluated predictions); the 83.3% gap is a coverage failure from dropped requests, whereas Category A syntax/enum errors occur when a model emits unapproved enum strings without grammar-constrained decoding.

Does this failure prove we must immediately fine-tune? No. We must first diagnose which failure categories produced this score.

---

The baseline selection cascade

Rather than treating adaptation as binary, evaluate candidate architectures progressively along the Baseline Selection Cascade:

The Baseline Selection Cascade

Evaluate progressively from Level 0 Zero-Shot to Level 3 Frontier Model before considering weight adaptation.

Read this diagram as text

Flowchart starting with Level 0 Zero-shot prompting, progressing to Level 1 Few-shot in-context learning, Level 2 RAG, and Level 3 Frontier generalist model. A gating decision evaluates whether any in-context level meets operational SLOs. If yes, the verdict is DO NOT TRAIN. Only if all fail and request volume justifies capital cost does the project proceed to candidate adapter fine-tuning.

1. Level 0 — Zero-Shot Prompting (baseline-0-zero-shot): The raw model with system instructions and schema definitions. Establishes the performance floor without demonstrations.
2. Level 1 — Few-Shot In-Context Learning (baseline-1-few-shot): Injects 3–5 representative, diverse exemplar pairs demonstrating input-to-output mappings across distinct task slices. Tests whether pattern demonstration fixes format and enum confusions.
3. Level 2 — Retrieval-Augmented Generation (baseline-2-rag): Dynamically retrieves relevant domain documents (e.g., operational runbooks, API contracts) into the context window based on the input incident query. Tests whether missing factual knowledge explains the gap.
4. Level 3 — Frontier Model Switch (baseline-3-frontier): Tests a higher-capacity open or commercial model (e.g., 70B parameters vs. 8B). Determines whether the failure is an intrinsic parameter capacity and multi-hop reasoning limit.
5. Candidate — Parameter-Efficient Fine-Tuning (LoRA / QLoRA): Modifies weights only if Levels 0–3 fail operational SLOs and volume amortizes the capital cost.

---

Four-category error taxonomy

Every observed error in LLM task execution falls into one of four orthogonal categories. Each has a specific, low-cost fix that does not require updating model weights.

Category	Root Cause & Failure Mode	Diagnostic Signal	Recommended Fix (Zero Weight Changes)
Category A: Syntax & Formatting	Missing required JSON keys, invalid enum values, markdown wrapper fences (```json), trailing commas.	schema_validity_rate < 100% across all slices.	Grammar-constrained decoding (e.g. Outlines, XGrammar, vLLM guided_json) or OpenAI/Anthropic Structured Outputs. Compiles JSON schema into a token-level finite state machine.
Category B: Domain Vocabulary & Facts	Hallucinated command flags, outdated endpoint names, unfamiliar error codes, missing runbook lookups.	High accuracy on standard_patterns, but severe failure on domain_heuristics lookups.	Retrieval-Augmented Generation (RAG). Ground generation with dynamically retrieved runbook chunks. Updating documents is instant; updating weights requires retraining.
Category C: Multi-Hop Reasoning & Policy	Misidentifying the root-cause service when errors cascade (e.g., mistaking an auth JWKS timeout for an API gateway bug); failing multi-condition threshold rules.	Fails on complex dependency chains despite having correct facts in context.	Chain-of-Thought (CoT) prompting with explicit scratchpad steps, or switching to a larger base model family (e.g. 70B parameter tier).
Category D: Adversarial & Safety Edge Cases	Suppressing human escalation during database split-brain, data corruption, or unauthenticated traffic spikes; applying automated remediation to an unstable cluster.	escalation_safety_recall < 95% on edge_and_safety slice.	Deterministic safety dominance. Enforce hard rule-based filters in code: if telemetry exhibits split-brain signatures, override model output to force requires_human_escalation: true and immediate_mitigation: noop.

Common mistake

Safety Recall Asymmetry: In safety-critical tasks, false negatives carry catastrophic consequences. In our incident triage benchmark, failing to escalate during an NVMe CRC corruption or Raft consensus split-brain risks irreversible data loss. Because neural networks remain probabilistic, fine-tuning alone can never guarantee 100% safety recall. Mission-critical boundaries must be guarded by deterministic software invariants, not stochastic weights.

---

In-context learning vs. weight updates: what the research proves

The instinct to fine-tune often stems from a misunderstanding of how LLMs absorb task patterns. Primary empirical literature demonstrates the power and boundaries of in-context learning:

1. Few-Shot Demonstrations as Task Specification: Brown et al. (2020) demonstrated in the foundational GPT-3 study that in-context learning achieves competitive performance with fine-tuned models on diverse benchmarks without a single gradient step. As model scale increases, the marginal gain from prompt demonstrations accelerates.

2. Mechanics of Demonstrations: Min et al. (2022) revealed a surprising finding in Rethinking the Role of Demonstrations: replacing ground-truth labels with completely random labels in few-shot exemplars barely degrades performance (a drop of only 0% to 5% across 12 different models). What few-shot prompting actually teaches the model is:

   • The input formatting and structure.
   • The input distribution (the domain style).
   • The valid output label space (the permitted enum set).

   If your model emits invalid enums or poor formatting, providing 3 fixed exemplars solves the issue immediately by delineating the label space.

3. Parametric vs. Non-Parametric Memory: Lewis et al. (2020) showed in the original RAG paper that parametric memory (weights) is inherently unsuited for precise, rapidly shifting factual lookups. Parametric factual recall suffers from hallucination and temporal staleness. Storing domain facts in external vector or BM25 indices and retrieving them at inference time outperforms parametric storage while allowing real-time knowledge updates.

4. Structured Outputs are Solved at the Decoder Level: Recent work on grammar-constrained decoding (Willard & Louf, 2023; Geng et al., 2025) proves that token-level masking guarantees syntactic schema conformance for any schema supported by the grammar engine — without fine-tuning. Fine-tuning a model solely to teach it JSON syntax is obsolete engineering.

---

Economic break-even derivation: the token amortization threshold

Even if an adapted model achieves marginally better latency or token economy than a long few-shot prompt, the economics of hosting often invalidate the investment.

Cost Formulation

Consider an operational horizon H = 12 months and monthly query volume V:

1. Monthly In-Context Prompting Cost (C_prompt):

   C_prompt(V) = V × ((L_in_prompt / 1000) × P_in + (L_out / 1000) × P_out)

2. Monthly Adapted Specialist Cost (C_adapt) — Two Distinct Deployment Regimes: Never double-count dedicated GPU instance rental and per-token API billing in the same equation. Depending on how you serve the adapter, cost follows one of two distinct regimes:

   • Regime 1: Self-Hosted Dedicated GPU (C_host = \$1,350.50/month, marginal token API fee = $0/req up to GPU throughput capacity):

     C_adapt_self(V) = C_host + (C_one_time / H)
     V*_self_hosted = (C_host + C_one_time / H) / C_prompt_req

   • Regime 2: Serverless / Multi-Tenant LoRA API (C_host = \$0/month, billed per token at C_adapt_req = \$0.000235/req):

     C_adapt_serverless(V) = (C_one_time / H) + V × ((L_in_adapt / 1000) × P_in + (L_out / 1000) × P_out)
     V*_serverless = (C_one_time / H) / ΔC_token

Where:

• C_host: Fixed monthly dedicated GPU hosting ($1,350.50/month for a single AWS/GCP instance running 730 hours × \$1.85/hr).
• C_one_time = C_data + C_train = \$3,007.40 (data curation $3,000.00 + GPU training compute $7.40), or \$250.62/month over H = 12 months.
• C_prompt_req: Full baseline prompt API cost per request ((900/1000)×\$0.0005 + (90/1000)×\$0.0015 = \$0.000585/req).
• ΔC_token: Variable token cost savings per request under per-token billing:

  ΔC_token = ((L_in_prompt - L_in_adapt) / 1000) × P_in = 0.700 × $0.0005 = $0.000350 per request

Worked Numerical Example

Using the verified parameters from labs/track3/01-baseline-decision/evaluate.py:

• Regime 1 (Self-Hosted Dedicated GPU):

  Fixed Monthly Overhead = $1,350.50 + ($3,007.40 / 12) = $1,601.12 / month
  V*_self_hosted = $1,601.1167 / $0.000585 ≈ 2,736,952 requests/month

• Regime 2 (Serverless / Multi-Tenant LoRA API with C_host = \$0):

  Fixed Monthly Overhead = $3,007.40 / 12 = $250.62 / month
  V*_serverless = $250.6167 / $0.000350 ≈ 716,048 requests/month

Note

Economic Reality Check: If your production service processes 5,000 incidents a month, the few-shot prompting baseline costs $2.93/month in token billing. Hosting a dedicated fine-tuned model costs $1,601.12/month (V*_self_hosted = 2,736,952 req/mo), and even free multi-tenant serverless LoRA (C_host = \$0) requires 716,048 req/mo just to amortize the $3,007.40 data curation and training investment. At 5,000 req/mo, fine-tuning is only justified when policy or security constraints (such as air-gapped on-premises deployment or strict PII residency) prohibit external APIs.

---

The four-gate decision engine

Before committing to dataset curation or training in Module 2, the team must pass through four sequential decision gates:

The Four-Gate Decision Engine

Evaluate sequentially through four gates before committing to weight adaptation: Quality baselines, Frontier generalists, Data stability, and Economic viability.

Read this diagram as text

Flowchart showing four sequential gates: Gate 1 (Quality Baseline), Gate 2 (Frontier Model), Gate 3 (Data Stability), and Gate 4 (Economic Amortization). Triggering any gate results in DO NOT TRAIN. Only passing all four gates leads to Proceed to Adaptation.

Detailed Gate Specifications

1. Gate 1: Baseline Quality Sufficiency
   • Condition: If baseline-1-few-shot or baseline-2-rag achieves coverage_rate == 1.00, schema_validity_rate == 1.00, joint_task_accuracy >= 0.75, and escalation_safety_recall >= 0.95.
   • Verdict: DO NOT TRAIN. Baseline prompt engineering satisfies production SLOs.
2. Gate 2: Frontier Generalist Feasibility
   • Condition: If an existing larger generalist model (e.g. 70B open instruct or commercial API) satisfies quality thresholds, and its monthly token cost is strictly less than dedicated adapter hosting (C_frontier(V) < C_host).
   • Verdict: DO NOT TRAIN. Model capacity solved the reasoning gap without custom weight maintenance.
3. Gate 3: Data Provenance & Domain Volatility
   • Condition: If verified, audited training pairs N < 250, or if the underlying API contracts and runbooks change faster than 20% per quarter.
   • Verdict: DO NOT TRAIN. Low data volume induces severe overfitting; high domain drift renders fine-tuned weights obsolete immediately.
4. Gate 4: Economic Amortization
   • Condition: If projected volume V < V*, where V* is the amortization threshold (2,736,952 req/mo self-hosted or 716,048 req/mo serverless), and no policy/air-gapped security mandate prohibits external APIs.
   • Verdict: DO NOT TRAIN. Monthly fixed infrastructure exceeds variable token savings.

Only when all four gates evaluate to proceed is model adaptation justified.

(Note: evaluate.py in Lab 01 automates Gate 1 Quality and Gate 4 Economics; Gate 2 Frontier Switch and Gate 3 Data Stability are human architectural review gates.)

---

Hands-on lab walkthrough: running the baseline evaluator

The complete evaluator, held-out test suite, and decision engine are provided in labs/track3/01-baseline-decision/. The lab requires only the Python standard library—no GPU, PyTorch, or third-party packages needed.

1. Inspect the Evaluation Fixture

Examine labs/track3/01-baseline-decision/held_out_tasks.json. Notice how each incident record contains:

• incident_id: Unique identifier (e.g. INC-2026-001).
• slice: Difficulty tier (standard_patterns, domain_heuristics, edge_and_safety).
• ground_truth: Exact targets for subsystem, severity, mitigation, and escalation.

2. Execute the Evaluator

Run the evaluator on the sample zero-shot predictions:

python3 labs/track3/01-baseline-decision/evaluate.py \
  --predictions labs/track3/01-baseline-decision/sample_predictions_zero_shot.json

Observe the output:

• The script detects that INC-2026-004 (non-escalation coverage failure) and INC-2026-012 (safety false negative) were omitted.
• It computes slice-level accuracy, identifying that edge_and_safety failed catastrophically (0.0%).
• It computes the economic break-even volume (V*_self_hosted = 2,736,952 and V*_serverless = 716,048) and triggers:

  DECISION GATE VERDICT:  >>> DO_NOT_TRAIN <<<
  Gate Triggered:         GATE_4_ECONOMIC_NON_VIABILITY

3. Test the Quality Gate Pass (Oracle Verification)

Run the evaluator against sample_predictions_ideal.json (an oracle test fixture containing exact ground_truth targets to verify Gate 1 logic):

python3 labs/track3/01-baseline-decision/evaluate.py \
  --predictions labs/track3/01-baseline-decision/sample_predictions_ideal.json

Because the oracle predictions achieve 100% coverage, 100% schema validity, 100% joint accuracy, and 100% escalation safety recall, Gate 1 triggers:

DECISION GATE VERDICT:  >>> DO_NOT_TRAIN <<<
Gate Triggered:         GATE_1_BASELINE_QUALITY_SUFFICIENT

When prompt engineering, grammar-constrained decoding, and RAG satisfy production requirements, Gate 1 halts unnecessary weight training.

---

Key takeaways

• Diagnose Before Training: Never fine-tune an LLM without categorizing errors into syntax, factual, reasoning, and safety buckets.
• Syntax is Not a Weight Problem: Schema violations are solved via token-level grammar-constrained decoding (e.g. Outlines, vLLM guided_json), not weight updates.
• Facts Belong in RAG: Factual domain knowledge changes faster than training cycles; keep changing runbooks in non-parametric retrieval memory.
• Safety Requires Determinism: Critical safety decisions (like escalating on split-brain hazards) must be dominated by hard-coded software invariants, not probabilistic outputs.
• The Economic Gate: If monthly query volume cannot amortize dedicated GPU instance hosting (V < V*), fine-tuning is an economic mistake.

---

Try it yourself

1. 01Run evaluate.py against sample_predictions_zero_shot.json. Identify which two task IDs were missing, inspect why INC-2026-004 and INC-2026-012 receive different safety classifications when omitted, and confirm the unit tests pass with python3 -m unittest discover -s labs/track3/01-baseline-decision/.

   Show a hint

   In evaluate.py, INC-2026-004 has requires_human_escalation: false (coverage failure, no escalation required), whereas INC-2026-012 has requires_human_escalation: true (safety false negative).

2. 02Simulate a Category A Format Violation & Repair: Create a modified prediction file from sample_predictions_zero_shot.json that corrupts primary_subsystem in INC-2026-001 from db_connection_pool to an invalid enum string db_pool, run evaluate.py to observe schema_validity_rate drop from 100.0% (10/10) to 90.0% (9/10), and then restore the valid enum.

   Show a hint

   In sample_predictions_zero_shot.json, all 10 present records are syntactically valid (10/10 = 100.0% schema validity; 10/12 = 83.3% coverage). Injecting an unapproved enum demonstrates how grammar-constrained decoding prevents Category A errors without weight updates.

3. 03Sensitivity Analysis on Break-Even Query Volume: Use evaluate.py with --dedicated-hosting-cost 650.00 (spot/reserved pricing) and --monthly-volume 2000000. Recompute V*_self_hosted = ($650.00 + $250.62) / $0.000585 and verify that 2 million requests/month exceeds the ~1,539,516 req/mo break-even threshold.

   Show a hint

   At C_host = $650.00/mo, total fixed monthly cost is $900.62/mo. Dividing by $0.000585/req yields V*_self_hosted = 1,539,516 requests/month.

---

Interview questions

Under what circumstances would you reject fine-tuning even if an unadapted 8B model achieves only 40% accuracy on your domain benchmark?

You reject fine-tuning if: (1) The failure modes are Category A (syntax/formatting) or Category B (factual lookups), which can be resolved via schema-constrained decoding and RAG; (2) Switching to a larger generalist model (e.g. 70B) via API satisfies accuracy at a lower total monthly cost than dedicated GPU hosting; (3) You have fewer than 250 audited examples or high domain volatility (>20% quarterly churn); or (4) Monthly query volume is below the token amortization break-even threshold V*, making dedicated adapter serving economically unjustified.

Why does Min et al. (2022) prove that few-shot in-context learning is not primarily teaching the model input-to-output reasoning?

Min et al. demonstrated that replacing ground-truth labels with randomly sampled incorrect labels in few-shot demonstrations causes almost no performance drop (0% to 5%) across diverse benchmarks. This proves that demonstrations primarily communicate the input distribution, formatting syntax, and valid output label space (the permitted enum set), rather than teaching the model novel semantic reasoning from scratch.

How does grammar-constrained decoding eliminate the need to fine-tune for JSON formatting?

Grammar-constrained decoding (e.g., Outlines, XGrammar, vLLM guided_json) compiles a JSON Schema into a finite-state machine or context-free grammar. At each autoregressive decoding step, the engine masks out all vocabulary tokens that would violate the grammar, reducing their logits to -infinity. The model is physically incapable of emitting invalid syntax or unapproved enums, achieving 100% syntactic validity without updating weights.

Why should high-consequence safety triggers (like split-brain escalation) never rely solely on fine-tuned weights?

Deep neural networks are probabilistic token predictors subject to calibration drift, edge-case hallucinations, and adversarial prompt injection. In safety-critical scenarios (e.g., stopping automated remediation during a distributed database partition), a single false negative causes catastrophic data loss. Critical boundaries must be governed by deterministic software rules (guardrails) that enforce human escalation regardless of model predictions.

---

Related lessons

• LLM Internals & Architecture · Reasoning models and test-time computationEvaluating Reasoning: Baselines, Verifiers, and Parsing Failures12 min
• Inference Optimisation & AI Infrastructure · Measure and Explain Inference PerformanceServing Workloads and Performance Metrics12 min

---

Sources and scope

• Brown et al. (2020) — Language Models are Few-Shot Learners, NeurIPS 2020. arXiv:2005.14165. Demonstrates few-shot in-context scaling and establishes that prompting can rival fine-tuning on knowledge tasks.
• Min et al. (2022) — Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?, EMNLP 2022. arXiv:2202.12837. Proves that demonstrations primarily condition the model on input format and label space rather than input-label mapping.
• Lewis et al. (2020) — Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, NeurIPS 2020. arXiv:2005.11401. Establishes RAG as the preferred mechanism for non-parametric factual recall over static weight storage.
• Willard & Louf (2023) — Efficient Guided Generation for Large Language Models, arXiv:2307.09702. Formulates token-level finite-state machine masking for guaranteed schema compliance.
• Geng et al. (2025) — JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Models, arXiv:2501.10868. Evaluates schema-constrained frameworks across 10,000 real-world schemas.

Scope and Data Boundaries:

• Measured Data: The 12 held-out task cases, ground-truth label annotations, deterministic scoring equations, and zero-shot baseline evaluations reported above were computed directly by labs/track3/01-baseline-decision/evaluate.py. The 12 cases represent a self-authored synthetic evaluation fixture designed to test multiple failure categories across syntax, factual knowledge, reasoning, and safety domains.
• Illustrative Data: Token pricing tiers ($0.0005 input, $0.0015 output per 1,000 tokens) and dedicated GPU hosting rates ($1.85/hr) represent representative cloud market rates; actual corporate contract pricing and serverless LoRA multi-tenant runtime economics vary. No phantom GPU benchmark measurements were invented.