Set a Baseline and Success Threshold

The one-sentence version

Before training an adapter or fine-tuning weights, you must freeze an immutable evaluation set, define multi-dimensional quality and regression thresholds, and commit to a formal decision record that halts the project if prompting, retrieval, or larger generalists already solve the problem.

Without pre-declared success thresholds, teams shift goalposts post-training, mistaking overfitting on a narrow prompt set for true domain competence.

---

The engineering discipline: defining thresholds before gradients

In traditional software engineering, test suites and Service Level Objectives (SLOs) are committed to version control before shipping code. In machine learning, teams frequently invert this discipline: they fine-tune a model, inspect cherry-picked generations, and retroactively invent benchmarks that justify the training run.

A Staff/Principal AI systems engineer establishes an immutable contract before training begins:

1. Freeze the held-out evaluation fixture: No example from the test set may ever enter training, validation, or prompt tuning sets.
2. Decompose aggregate metrics into slices: Aggregate accuracy conceals catastrophic safety failures on rare edge cases.
3. Set non-negotiable safety recall thresholds: False negatives on mission-critical actions (e.g. failing to escalate a database partition) carry severe asymmetric costs.
4. Guard against catastrophic forgetting: Fine-tuning on domain data must not silently erode the base model’s general reasoning or instruction-following capabilities.
5. Formalize the Decision Record: A structured, reviewable document that governs whether the project ships the prompting baseline, proceeds to training, or holds.

(Note: In our production harness labs/track3/01-baseline-decision/evaluate.py, Gate 1 Quality and Gate 4 Economics are deterministically automated in code, while Gate 2 Frontier Feasibility and Gate 3 Data Stability are human architectural review gates.)

---

1. The four-dimension success threshold matrix

A production model cannot be judged on raw accuracy alone. The evaluation contract establishes four independent dimensions that every candidate architecture must satisfy:

Evaluation Dimension	Primary Metric	Minimum Promotion Threshold	Engineering Rationale
1. Domain Task Quality	schema_validity_rate	= 1.00 (100%)	Syntactically invalid JSON breaks downstream automation pipelines.
	joint_task_accuracy	≥ 0.75 (75%)	Exact match across subsystem, severity, and mitigation action.
	escalation_safety_recall	≥ 0.95 (95%)	Missing a required human escalation during state corruption is catastrophic.
	escalation_precision	≥ 0.70 (70%)	Prevents alert fatigue and unnecessary human on-call pages.
2. Capability Regression	MMLU / ARC benchmark delta	Point Δ ≥ −3% to −5%	Verifies the adapter has not induced catastrophic forgetting of reasoning.
3. Serving Performance	P99 Time-to-First-Token	≤ 400 ms	Ensures interactive operator latency during incident triage.
	P99 End-to-End Latency	≤ 800 ms	End-to-end execution budget for automated remediation triggers.
4. Economic Amortization	Break-even volume V*	Monthly Volume V ≥ V*	Serving dedicated weights must be cheaper than API tokens at production scale.

The Joint Accuracy Standard

In our incident triage task, partial accuracy is deceptive. A model that correctly identifies subsystem: consensus_engine and severity: critical but recommends restart_cluster_nodes instead of isolate_split_brain_partition will cause catastrophic state corruption. Joint accuracy requires every critical field to be simultaneously correct.

---

2. Behavioral slicing: exposing what averages hide

Aggregate accuracy is the single most dangerous metric in applied AI. As Ribeiro et al. (2020) demonstrated in Beyond Accuracy: Behavioral Testing of NLP Models with CheckList, high aggregate performance frequently masks severe systematic defects on critical subgroups.

In our production incident triage evaluation fixture (labs/track3/01-baseline-decision/held_out_tasks.json), the 12 held-out cases are partitioned into three orthogonal slices:

Held-Out Evaluation Set (N = 12 Cases)
├── Slice 1: standard_patterns (4 cases, 33%)
│   └── Unambiguous single-service failures (e.g. connection pool exhaustion, memory leak).
├── Slice 2: domain_heuristics (4 cases, 33%)
│   └── Multi-service cascading errors requiring internal runbook knowledge (e.g. JWKS auth timeout).
└── Slice 3: edge_and_safety (4 cases, 33%)
    └── High-consequence adversarial states (e.g. Raft split-brain, NVMe CRC corruption).

The Slicing Breakdown in Action

When evaluating our unadapted zero-shot baseline fixture (sample_predictions_zero_shot.json) against the sliced test fixture using evaluate.py:

Evaluation Slice	Number of Cases	Coverage	Schema Validity (Evaluated)	Joint Accuracy	Escalation Safety Recall	Production Assessment
standard_patterns	4 cases	75.0% (3/4)	100.0% (3/3)	75.0% (3/4)	100.0% (vacuous: 0 ground truth)	INC-2026-004 omitted (non-escalation); 3/3 evaluated pass.
domain_heuristics	4 cases	100.0% (4/4)	100.0% (4/4)	25.0% (1/4)	100.0% (vacuous: 0 ground truth)	Fails cross-service root-cause attribution.
edge_and_safety	4 cases	75.0% (3/4)	100.0% (3/3)	0.0% (0/4)	0.0% (0/4)	CATASTROPHIC FAILURE: Zero safety escalations (INC-2026-012 omitted).
Aggregate Overview	12 cases	83.3% (10/12)	100.0% (10/10)	33.3% (4/12)	0.0% (0/4)	Unshippable due to safety slice collapse and 83.3% coverage.

The harness surfaces an immediate warning: 2 missing predictions (INC-2026-004 and INC-2026-012). Because INC-2026-012 is a critical safety incident requiring human escalation, its omission directly counts as an escalation false negative (while INC-2026-004 requires no escalation and counts as a coverage failure). On the remaining three edge cases, the model emitted automated restart commands during active database partitions. Overall safety recall collapsed to 0.0% (with 0 predicted positives across all 12 tasks, making escalation precision N/A).

Visualizing the Evaluation Slicing Architecture

Evaluation Slicing and Multi-Dimensional Gate Hierarchy

Read this diagram as text

Diagram showing the three evaluation slices: standard patterns, domain heuristics, and safety edge cases, with their respective accuracy and safety recall targets.

---

3. Safety recall asymmetry: the cost of false negatives

In statistical evaluation, precision and recall are often balanced using harmonic means such as F1 score. In production systems engineering, this balance is fundamentally broken.

As Amodei et al. (2016) noted in Concrete Problems in AI Safety, safety-critical AI systems operate under severe cost asymmetry:

Cost(False Negative) >>> Cost(False Positive)

In our operational incident routing architecture:

• False Positive (Spurious Escalation): The model mistakenly pages the human on-call engineer for a routine Redis eviction.
  • Consequence: An engineer spends 5 minutes reviewing the alert. The system remains operational.
• False Negative (Missed Escalation): The model attempts automated node termination during an active Raft split-brain partition instead of halting and paging a human.
  • Consequence: The surviving minority partition loses quorum; uncommitted transactions are lost; database state diverges permanently.

The F1 Score Fallacy in AI Safety

Never optimize an AI safety policy for symmetric F1 score. A model with 90% precision and 80% recall has an F1 score of 0.847, yet it misses 20% of catastrophic incidents. In our evaluation protocol, escalation_safety_recall >= 0.95 is an unyielding requirement, whereas precision threshold is relaxed to 0.70.

Missing Predictions as Safety Violations

A robust scoring harness must never impute or ignore missing predictions. In labs/track3/01-baseline-decision/evaluate.py:

# Missing prediction handling in evaluate.py (lines 154-161)
if task_id not in predictions:
    missing_predictions.append(task_id)
    if gt_rec["requires_human_escalation"]:
        escalation_false_negatives += 1
    else:
        escalation_true_negatives += 1
    continue

If a model times out, crashes, or omits a prediction for an incident that required human intervention, the scorer immediately registers an escalation false negative, directly penalizing safety recall.

---

4. Guarding against catastrophic forgetting: the regression gate

When adapting a base model on domain data, gradient updates optimize for domain loss at the expense of pretraining representations. Kirkpatrick et al. (2017) and French (1999) demonstrated that this catastrophic forgetting degrades general reasoning, commonsense deduction, and instruction adherence.

To prevent shipping an adapter that excels at JSON syntax but degrades in general capability, production teams enforce an immutable Regression Gate:

Δ_benchmark = Score_adapted(Benchmark) - Score_base(Benchmark)
Threshold: Point Δ_benchmark ≥ -3.0% (MMLU) and ≥ -5.0% (ARC / HumanEval)

The Frozen General Benchmark Suite

1. Freeze the evaluation suite at project initiation: Select standard reasoning benchmarks (e.g. MMLU 57-subject subset, ARC-Challenge, HumanEval Pass@1).
2. Deterministic execution: Run both the unadapted base checkpoint and the fine-tuned candidate with greedy decoding (temperature = 0.0).
3. Paired bootstrap confidence intervals: Compute 10,000 bootstrap iterations over the difference in per-example correctness. A candidate is rejected if its point degradation falls below threshold (Δ < -3.0%), or if the paired bootstrap 95% confidence interval upper bound is less than zero, confirming statistically significant capability regression.

---

5. Latency and economic amortization targets

A candidate model that achieves 100% accuracy and zero regression remains unviable if it cannot meet latency SLOs or amortize hosting expenses.

5.1 The Tail-at-Scale Serving Budget

Dean & Barroso (2013) demonstrated in The Tail at Scale that aggregate mean latency hides tail bottlenecks that degrade distributed systems. Our automated incident triage pipeline enforces strict P99 latency bounds:

P99 Time-to-First-Token (TTFT) ≤ 400 ms
P99 End-to-End Latency (E2E)    ≤ 800 ms
Sustained Concurrency           ≥ 50 requests/second without queuing collapse

5.2 The Break-Even Sensitivity Model

Recall from Lesson 1 the two economic amortization regimes (never double-counting dedicated GPU rental and per-token billing):

Self-Hosted Dedicated GPU (marginal token API cost = $0):
V*_self_hosted = (C_host + C_train / H) / C_prompt_req

Serverless / Multi-Tenant LoRA API (C_host = $0, billed per token):
V*_serverless  = (C_train / H) / ΔC_token

Where C_host is monthly dedicated GPU hosting, C_train is one-time data curation and training compute ($3,007.40), H = 12 months, C_prompt_req = $0.000585/req is baseline prompt cost, and ΔC_token = $0.000350/req is the variable token savings per request under per-token billing.

In production planning, Staff Engineers generate a Break-Even Sensitivity Matrix across hosting architectures:

Serving Architecture	Monthly Fixed Cost (C_host)	Amortized Training (C_train / 12)	Total Fixed Overhead	Marginal Cost Comparison	Break-Even Volume V*	Viability at V = 5,000 req/mo
Dedicated Cloud GPU (A10G/L4)	$1,350.50 / mo	$250.62 / mo	$1,601.12 / mo	vs $0.000585/req prompt API	2,736,952 req/mo	❌ NON-VIABLE
Reserved 1-Year GPU Instance	$850.00 / mo	$250.62 / mo	$1,100.62 / mo	vs $0.000585/req prompt API	1,881,396 req/mo	❌ NON-VIABLE
Multi-Tenant Serverless LoRA	$0.00 / mo (shared base)	$250.62 / mo	$250.62 / mo	vs $0.000350/req token savings	716,048 req/mo	❌ NON-VIABLE at 5k requests
Air-Gapped On-Premises Mandate	Policy requirement	$250.62 / mo	Required by security	External cloud API prohibited	Policy-mandated	✅ PROCEED TO MODULE 2

Unless monthly query volume exceeds the break-even threshold V* (2,736,952 req/mo self-hosted or 716,048 req/mo serverless), or corporate data-residency policy prohibits external APIs, fine-tuning is economically unjustified.

---

6. The Module 1 exit artifact: the Ship / Train / Hold decision record

Module 1 culminates in an immutable, signed Decision Record. This document acts as the formal architectural gate between problem diagnosis and dataset engineering.

evaluate.py --predictions pred.json --json-output decision_record.jsonIf verdict != ‘PROCEED_WITH_ADAPTATION’, pipeline blocks progression to Module 2 training scripts.

The Module 1 Decision Record Lifecycle

Read this diagram as text

Flowchart showing the progression from held-out evaluation through the four-gate decision engine to the formal signed decision record.

Production Decision Record Specification

Every Module 1 review generates a signed architectural Markdown record synthesized from the harness’s JSON output:

# Ship / Train / Hold Decision Record

## Incident Triage Specialist — Module 1 Exit Review

### 1. Context & Scope

- **Task:** Production Incident Triage & Operational Runbook Routing
- **Evaluation Fixture:** `labs/track3/01-baseline-decision/held_out_tasks.json` (**12-case self-authored synthetic evaluation fixture**, v1.0.0, across 3 slices)
- **Base Model Evaluated:** `Qwen/Qwen2.5-7B-Instruct` (Alternative: `meta-llama/Llama-3.1-8B-Instruct`)
- **Evaluation Engine:** Deterministic evaluation harness `labs/track3/01-baseline-decision/evaluate.py`

### 2. Multi-Candidate Evaluation Results

_(Note: Baseline 0 is scored directly by `evaluate.py` on `sample_predictions_zero_shot.json`; Baselines 1–3 are illustrative hypothetical candidate comparisons showing how progressive non-training baselines are evaluated.)_

| Architecture                                 | Coverage       | Schema Validity (Evaluated) | Joint Accuracy | Safety Recall        | P99 E2E Latency       | Monthly Cost @ 5k Reqs |
| :------------------------------------------- | :------------- | :-------------------------- | :------------- | :------------------- | :-------------------- | :--------------------- |
| **Baseline 0: Zero-Shot Fixture (Scored)**   | 83.3% (10/12)  | 100.0% (10/10)              | 33.3%          | 0.0% (0/4 on edge)   | 380 ms (illustrative) | $1.30 / mo             |
| **Baseline 1: Few-Shot (Illustrative)**      | 100.0% (12/12) | 100.0% (12/12)              | 58.3%          | 50.0% (2/4 on edge)  | 540 ms (illustrative) | $2.93 / mo             |
| **Baseline 2: RAG Grounding (Illustrative)** | 100.0% (12/12) | 100.0% (12/12)              | 66.7%          | 75.0% (3/4 on edge)  | 720 ms (illustrative) | $3.45 / mo             |
| **Frontier API (70B, Illustrative)**         | 100.0% (12/12) | 100.0% (12/12)              | 83.3%          | 100.0% (4/4 on edge) | 760 ms (illustrative) | $18.20 / mo            |

### 3. Decision & Gate Verdict

- **VERDICT:** **DO NOT TRAIN (AT 5,000 REQ/MO WITHOUT ON-PREMISES POLICY MANDATE)**
- **PRIMARY GATE TRIGGERED:** **`GATE_4_ECONOMIC_NON_VIABILITY`**
- **ECONOMIC SUMMARY:** Self-hosted Dedicated GPU break-even volume `V*_self_hosted = 2,736,952 requests/month`. Serverless / Multi-Tenant LoRA API break-even volume `V*_serverless = 716,048 requests/month`. At `V = 5,000 requests/month`, dedicated fine-tuning inflates monthly infrastructure expenditure from $18.20/month (Frontier API) to $1,601.12/month (Dedicated GPU Instance). Training at `5,000 req/mo` is only justified when **policy/security constraints** (such as air-gapped on-premises deployment or strict PII residency) prohibit external Frontier APIs.
- **TECHNICAL CONTINGENCY:** If corporate policy mandates air-gapped on-premises execution (prohibiting external Frontier APIs) OR if query volume scales past `716,048 req/mo` (serverless) / `2,736,952 req/mo` (self-hosted), Gate 4 clears and the project promotes to **Module 2: Trustworthy Training Data**.

### 4. Architectural Sign-Off

- **Lead AI Systems Engineer:** [UNSIGNED TEMPLATE — PENDING HUMAN ARCHITECTURAL SIGN-OFF]
- **SRE & Production Operations Lead:** [UNSIGNED TEMPLATE — PENDING HUMAN ARCHITECTURAL SIGN-OFF]

---

Key takeaways

• Pre-Commitment Discipline: Success thresholds, frozen test sets, and decision gates must be locked in version control before fine-tuning begins.
• Aggregate Accuracy Masks Failures: Slicing evaluation into standard, heuristic, and safety-critical subsets reveals localized catastrophic failures.
• Safety Recall Asymmetry: In high-consequence domains, missing a critical escalation carries far greater cost than a spurious page; enforce safety recall >= 95%.
• Catastrophic Forgetting Guardrails: Monitor frozen general benchmarks (MMLU, ARC) to verify adapter training does not silently erode general capabilities.
• The Formal Decision Record: Module 1 terminates in a signed Ship/Train/Hold artifact that prevents unjustified GPU training expenditures.

---

Try it yourself

1. 01Run evaluate.py against both sample_predictions_zero_shot.json and sample_predictions_ideal.json. Compare the per-slice accuracy across standard_patterns, domain_heuristics, and edge_and_safety, and verify which gate each file triggers.

   Show a hint

   Notice that zero-shot triggers GATE_4_ECONOMIC_NON_VIABILITY due to volume V < V*, while ideal triggers GATE_1_BASELINE_QUALITY_SUFFICIENT.

2. 02Write a custom prediction JSON file that achieves 100% schema validity and 100% accuracy on standard and heuristic slices, but fails a single safety escalation in edge_and_safety. Run evaluate.py and explain why Gate 1 rejects promotion.

   Show a hint

   In evaluate.py, escalation_safety_recall must be strictly >= 0.95. A single missed escalation drops recall to 75%, failing Gate 1.

3. 03Perform a break-even sensitivity calculation. Prove why a monthly volume of 250,000 requests cannot amortize fine-tuning even if dedicated hosting is completely free (C_host = $0), and find the minimum monthly volume required for free serverless LoRA to break even.

   Show a hint

   At 250k volume, token savings is 250,000 × $0.000350 = $87.50/month, which fails to cover the $250.62/month amortized training cost. Solve V* = $250.6167 / $0.000350 ≈ 716,048 requests/month.

---

Interview questions

Why is aggregate accuracy an insufficient metric for evaluating domain-specific fine-tuning models in mission-critical environments?

Aggregate accuracy averages performance across all test instances, allowing high accuracy on routine, frequent cases (e.g. 95% on standard syntax) to obscure catastrophic failure on rare, high-consequence edge cases (e.g. 0% recall on distributed split-brain). In mission-critical environments, performance must be evaluated across behavioral slices (CheckList methodology) with explicit safety recall thresholds on adversarial subsets.

What is catastrophic forgetting, and how do production AI teams construct regression guardrails to detect it during fine-tuning?

Catastrophic forgetting occurs when gradient updates during domain fine-tuning overwrite foundational representations supporting general reasoning, mathematics, or instruction-following. Teams construct regression guardrails by freezing an immutable suite of broad benchmarks (e.g. MMLU, ARC, HumanEval) at project start and evaluating both the base model and fine-tuned adapter on identical prompts, rejecting any candidate where capability drops exceed 3% to 5%.

In our incident triage domain, why is missing prediction handling in the evaluation harness critical for safety recall accounting?

If an evaluation harness silently drops or imputes missing predictions (e.g. when a model times out or crashes on malformed JSON), a failure to triage a high-severity incident is ignored. In our evaluation protocol, any missing prediction on an incident that required human escalation is explicitly scored as a False Negative, directly penalizing safety recall and preventing fragile models from passing quality gates.

Under what exact condition does the four-gate decision engine conclude that an AI team should NOT fine-tune a model even when the prompting baseline fails quality targets?

When Gate 4 (Economic Amortization) fails. Even if prompt engineering achieves only 40% accuracy, if projected monthly query volume cannot amortize the fixed costs of dedicated GPU hosting and custom weight maintenance (V < V*), and a larger frontier generalist API can solve the problem at lower total monthly cost without dedicated hardware, fine-tuning is an economic mistake.

---

Related lessons

• Fine-Tuning & Domain-Specific Models · Decide Whether to TrainDiagnose the Gap Before Changing Weights12 min
• Fine-Tuning & Domain-Specific Models · Decide Whether to TrainSelect a Base Model for the Task13 min

---

Sources and scope

• Ribeiro et al. (2020), Beyond Accuracy: Behavioral Testing of NLP Models with CheckList, ACL 2020 — Subgroup behavioral analysis, orthogonal test slices, and aggregate metric failure modes.
• Amodei et al. (2016), Concrete Problems in AI Safety, arXiv:1606.06565 — Asymmetric cost of false negatives vs false positives in critical decision automation.
• Kirkpatrick et al. (2017), Overcoming catastrophic forgetting in neural networks, PNAS — Mechanics of representation overwriting during domain adaptation and regression mitigation.
• French (1999), Catastrophic forgetting in connectionist networks, Trends in Cognitive Sciences — Foundational cognitive and neural network theory on stability-plasticity dilemma.
• Dean & Barroso (2013), The Tail at Scale, Communications of the ACM — High-percentile tail latency (P99/P99.9) behavior in distributed systems and serving SLO design.

Local Evidence Notice: The held-out evaluation fixture (held_out_tasks.json), deterministic evaluation script (evaluate.py), and test suite (test_evaluate.py) in labs/track3/01-baseline-decision/ provide verified, reproducible ground-truth measurements. Economic parameters reflect standard cloud market tiers for educational comparison. No synthetic benchmarks are invented.