Split, Deduplicate, and Audit the Dataset

Section 1: The Practical Outcome First

In machine learning tutorials, splitting data is treated as a trivial one-liner: train_test_split(data, test_size=0.2). In production domain adaptation, applying a random independent and identically distributed (i.i.d.) split is an architectural disaster.

When infrastructure alerts fail, automated monitoring tools emit repetitive storms of near-identical messages. For example, a single Redis failover can generate 500 alert tickets over 20 minutes across different application pods. If you split this dataset randomly, 400 tickets land in the training set and 100 in the test set.

Your evaluation harness reports an astonishing 99.4% accuracy. In reality, your model has simply memorized specific pod names and timestamp strings. The moment a novel database failure occurs in production, accuracy collapses to 22.0%.

Look at the difference between naive random splitting and disciplined dataset curation:

[Naive Approach: Random i.i.d. Split on Raw Traces]
Raw Dataset: 10,000 incident tickets (including 2,400 repeated cron & alert storms)
Split: Random 80/20
Train Set: 8,000 examples (contains 1,920 near-duplicates of test tickets)
Test Set:  2,000 examples (contains 480 near-duplicates of train tickets)
Test Accuracy: 99.4% (Deceptive! 80% duplicate leakage)
Production Day 1: Model fails on novel network partition (22.0% true accuracy)

[Production Approach: MinHash Deduplication + Group-Aware Split + Decontamination]
Step 1: MinHash + LSH Near-Duplicate Pruning -> 10,000 rows reduced to 3,200 canonical examples
Step 2: Group-Aware Split on cluster_id -> Zero cluster overlap between Train and Test
Step 3: 8-gram Benchmark Decontamination -> Purged 4 overlapping evaluation tasks
Evaluated Test Accuracy: 84.2% (Honest! Zero leakage)
Production Day 1: Model maintains 83.8% accuracy on unseen infrastructure clusters

---

Section 2: Splitting Strategies and Optimistic Leakage

To evaluate whether a fine-tuned specialist truly generalizes, the validation split must represent unobserved scenarios rather than rephrased copies of training rows.

Comparison of Random i.i.d. Splitting versus Group-Aware and Temporal Splitting

Read this diagram as text

Random splitting scatters related records from the same cluster into both train and test partitions, causing optimistic leakage. Group-aware splitting isolates entire entity clusters. Temporal splitting trains on past time windows and tests strictly on future windows.

1. Group-Aware (Entity-Level) Splitting

In domain systems, data points cluster naturally around parent entities:

• Incident Triage: Group by incident_ticket_id, service_topology_cluster, or root_cause_incident_uuid.
• Customer Support: Group by user_account_id or organization_domain.
• Code Generation: Group by repository_name or library_dependency.

When splitting, all records belonging to group G_k are routed exclusively to the training partition or the testing partition. If Cluster kafka-broker-prod is in the test set, the model has never observed a single alert from that specific broker during training, proving it can diagnose Kafka partitions from first principles.

2. Temporal Splitting

Software systems evolve: deployment pipelines push new container images, database schemas gain columns, and network topologies shift.

A random split samples uniformly across time, allowing the model to train on records from Thursday, evaluate on records from Wednesday, and train on records from Friday. In production, models can only make predictions on future time horizons.

A Temporal Split sets a strict timestamp cutoff T_cutoff:

• Train Window: All records where timestamp < T_cutoff (e.g., Months 1–5).
• Validation Window: All records where T_cutoff <= timestamp < T_eval (Month 6).
• Test Window: All records where timestamp >= T_eval (Month 7).

This evaluates whether the specialist model degrades when infrastructure changes or new microservices are introduced.

---

Section 3: Exact & Near-Duplicate Deduplication (MinHash & LSH)

Duplication is not merely an inefficiency; it degrades model capability. Lee et al. (2022) and Carlini et al. (2022) demonstrated that duplicated sequences are memorized at rates disproportionately higher than unique sequences, reducing output diversity and increasing susceptibility to prompt injection and private data extraction.

3.1 Exact Deduplication

Exact deduplication normalizes whitespace, strips ANSI escape sequences, converts text to lowercase, and computes an SHA-256 hash:

content_hash = hashlib.sha256(normalized_text.encode('utf-8')).hexdigest()

If content_hash has already been seen in the dataset registry, the example is discarded. Exact hashing prunes identical bot messages and duplicate log emissions, but fails when timestamps, IP addresses, or minor log numbers change.

3.2 Near-Duplicate Deduplication: MinHash

To catch near-duplicates (such as two alerts differing only by host pg-01 vs pg-02), we use Jaccard Similarity over k-shingles (word n-grams):

J(A, B) = | S(A) intersect S(B) | / | S(A) union S(B) |

Comparing all pairs across N = 50,000 documents requires N(N-1)/2 ≈ 1.25 × 10^9 pairwise comparisons (O(N^2)), which stalls data pipelines.

MinHash (Broder, 1997) maps each document’s shingle set to a compact signature of M integers (typically M = 64 or 128):

1. Define M hash functions: h_i(x) = (a_i * x + b_i) mod p.
2. For each hash function, find the minimum hash across all shingles in document D:

   s_i(D) = min_{x in S(D)} h_i(x)

3. The Fundamental MinHash Theorem: The probability that two documents have the identical minimum hash equals their true Jaccard similarity:

   Pr[ s_i(A) == s_i(B) ] = J(A, B)

4. Estimate Jaccard similarity by counting matching signature positions:

   J_est(A, B) = (1 / M) * sum_{i=1}^M [ s_i(A) == s_i(B) ]

3.3 Locality Sensitive Hashing (LSH) Banding

To eliminate O(N^2) comparisons completely, partition the M signature integers into b bands of r rows (M = b * r):

• Each band of r integers is hashed into a bucket table.
• If two documents share identical hashes in at least one band, they are flagged as a candidate near-duplicate pair.
• The probability that two documents with true Jaccard similarity s collide in at least one band is:

  P(collision) = 1 - (1 - s^r)^b

With M = 64 hashes, partitioned into b = 16 bands of r = 4 rows:

• If similarity s = 0.80: P = 1 - (1 - 0.80^4)^16 = 1 - (1 - 0.4096)^16 = 1 - (0.5904)^16 = 0.99978 (99.98% discovery rate).
• If similarity s = 0.20: P = 1 - (1 - 0.20^4)^16 = 1 - (1 - 0.0016)^16 = 0.0253 (2.5% false-positive candidate rate).

LSH isolates candidate near-duplicates in O(N) linear time.

---

Section 4: Benchmark Decontamination

Evaluation leakage does not just occur between training and test splits; it frequently occurs between external training corpora and your standardized evaluation benchmarks.

Dodge et al. (2021) demonstrated that public datasets (such as the C4 web corpus) contain significant benchmark text contamination across standard NLP evaluation sets (such as GLUE, SuperGLUE, and generation tasks), motivating standard n-gram decontamination checks across modern benchmarks (including GSM8K and HumanEval).

The N-Gram Decontamination Protocol

To ensure your fine-tuned model cannot cheat by memorizing evaluation tasks:

1. Extract all n-grams (typically n = 8 or n = 13 consecutive words) from your held-out evaluation tasks into an in-memory hash set.
2. During the data curation pipeline, scan every incoming training candidate against the evaluation n-gram set.
3. If a training candidate contains an exact 8-gram match with any benchmark prompt or test solution, flag the record:

[DECONTAMINATION BREACH DETECTED]
Training Doc ID: train-ops-9182
Benchmark ID:    eval-bench-slice-03
Overlapping 8-gram: "kubernetes pod crashloopbackoff due to oom killed exit"
Action: Document purged from training corpus.

---

Section 5: Dataset Auditing and Slice Imbalance

A dataset that is deduplicated and split cleanly can still fail if class and operational slice distributions are severely skewed.

In operational incident management:

• P3 / Routine alerts (minor latency spikes, certificate warnings) comprise 80% to 90% of raw logs.
• P1 / Critical split-brain & safety failures comprise less than 3% of raw logs.

The tradeoff

The Majority-Class Collapse: If trained on raw unbalanced distributions, the model achieves a deceptively low cross-entropy loss by defaulting to predict "severity": "P3" on every single input. When a true P1 database split-brain occurs, the model fails to trigger executive safety escalations.

The Stratification Audit Table

Before launching a fine-tuning run, audit your partitioned splits against target slice representation thresholds (ensuring every partition column sums to exactly 100.0%):

Slice Category	Raw Population %	Target SFT Training %	Target Test %	Audit Action
P1 - Critical / Split-Brain	1.2%	20.0%	20.0%	Up-sample high-signal canonical P1 traces; synthesize teacher variations
P2 - Service Degraded	18.8%	30.0%	30.0%	Deduplicate repetitive cron alerts
P3 - Routine Telemetry	79.0%	40.0%	35.0%	Down-sample majority chatter using LIMA quality filter
Malformed / Missing Data	1.0%	10.0%	15.0%	Inject explicit malformed telemetry edge cases
Total Partition Sum	100.0%	100.0%	100.0%	Verified closed partition

---

Section 6: Worked Code Example & Verifier Walkthrough

The companion script scripts/verify-dedup-split.py implements the complete deduplication, MinHash LSH indexing, group-aware splitting, and decontamination pipeline.

6.1 MinHash Signature Generation

def compute_minhash_signature(self, shingles: Set[str]) -> List[int]:
    signature = [self.prime] * self.num_hashes
    for shingle in shingles:
        shingle_int = int(hashlib.md5(shingle.encode('utf-8')).hexdigest()[:8], 16)
        for i in range(self.num_hashes):
            h = (self.a_coeffs[i] * shingle_int + self.b_coeffs[i]) % self.prime
            if h < signature[i]:
                signature[i] = h
    return signature

6.2 Group-Aware Splitting

def group_aware_split(
    records: List[Dict[str, Any]],
    group_key: str = "cluster_id",
    train_ratio: float = 0.8,
) -> Tuple[List[Dict[str, Any]], List[Dict[str, Any]]]:
    grouped = defaultdict(list)
    for r in records:
        grouped[r[group_key]].append(r)

    group_keys = list(grouped.keys())
    random.shuffle(group_keys)

    train_records, test_records = [], []
    target_train_count = int(len(records) * train_ratio)

    for gk in group_keys:
        if len(train_records) < target_train_count:
            train_records.extend(grouped[gk])
        else:
            test_records.extend(grouped[gk])

    return train_records, test_records

6.3 Running the Verifier

Run the verifier script to execute the MinHash LSH index, group-aware split assertions, and benchmark decontamination check:

python3 scripts/verify-dedup-split.py

Expected output:

=== Running Topic 06: Split, Deduplication & Audit Verifier ===
[*] True Jaccard (doc1 vs doc2 near-duplicate): 0.385
[*] True Jaccard (doc1 vs doc3 distinct):        0.000
[*] Estimated MinHash Jaccard (doc1 vs doc2):   0.469
[*] Estimated MinHash Jaccard (doc1 vs doc3):   0.000
[*] LSH Discovered Candidate Near-Duplicates: {('doc1', 'doc2')}
[+] Test 1 (MinHash & LSH Near-Duplicate Detection) Passed.
[*] Train clusters: {'cluster-db-pool', 'cluster-auth-leak'}
[*] Test clusters:  {'cluster-kafka-lag'}
[*] Cluster Overlap: set()
[+] Test 2 (Group-Aware Splitting) Passed.
[*] Benchmark Decontamination Overlaps Detected: 1
    Overlapping n-gram: 'kubernetes pod crashloopbackoff due to oom'
[+] Test 3 (Benchmark Decontamination Audit) Passed.
=== All Topic 06 Verifications Passed Cleanly ===

---

Key takeaways

• Random i.i.d. splitting on raw telemetry causes severe optimistic leakage when repeated alert storms scatter identical failures across train and test.
• Group-aware splitting routes all records from a shared cluster or ticket entity exclusively to one side of the partition, guaranteeing zero entity leakage.
• Temporal splitting sets a historical cutoff, evaluating model resilience against operational drift and infrastructure evolution.
• MinHash estimates Jaccard similarity in constant signature space; LSH banding discovers candidate near-duplicates in linear time without O(N^2) pairwise comparisons.
• Benchmark decontamination scans training candidates for n-gram overlap against held-out evaluation tasks to prevent test set memorization.
• Stratification audits prevent majority-class collapse, ensuring rare critical safety escalations (P1) maintain at least 20-25% representation in SFT training.

Try it yourself

1. 01A team splits 10,000 incident traces randomly and observes 98% test accuracy, but in production the model fails on new service outages. Explain why group-aware splitting reveals true generalization.

   Show a hint

   In a random split, identical alert storms leak across train and test, allowing the model to achieve high test scores by memorizing specific pod IDs. Group-aware splitting isolates entire services.

2. 02Given M=64 MinHash functions divided into b=16 bands of r=4 rows, calculate the probability of discovering a candidate pair with true Jaccard similarity s=0.70.

   Show a hint

   P = 1 - (1 - s^r)^b = 1 - (1 - 0.70^4)^16 = 1 - (1 - 0.2401)^16 = 1 - (0.7599)^16 ≈ 0.987 (98.7%).

3. 03Why does training on raw operational logs without stratification cause the model to fail on critical P1 safety escalations?

   Show a hint

   P3 routine logs comprise over 80% of raw production logs. The loss minimizer learns to predict P3 on almost all inputs, starving the gradients needed for rare P1 safety rules.

4. 04Explain how an 8-gram containment scan prevents benchmark contamination between an SFT training corpus and an automated evaluation suite.

   Show a hint

   If 8 consecutive words match an evaluation question or solution, the training candidate has memorized the benchmark task and must be purged.

Interview questions

What is the difference between exact deduplication and near-duplicate deduplication in data engineering for LLMs?

Exact deduplication uses cryptographic hashes (SHA-256) on normalized text to catch identical strings. Near-duplicate deduplication uses MinHash and Locality Sensitive Hashing (LSH) over k-shingles to detect documents that share high Jaccard similarity even when timestamps, hostnames, or formatting tokens differ.

How does Locality Sensitive Hashing (LSH) reduce the computational complexity of duplicate detection from O(N^2) to linear time?

Instead of comparing all N*(N-1)/2 document pairs, LSH divides the M-dimensional MinHash signatures into b bands of r rows and hashes each band chunk into hash buckets. Documents that collide in the same bucket in any band become candidate pairs. This bounds comparison to candidates sharing buckets, operating in sub-quadratic O(N) time.

Why is temporal splitting preferred over random splitting for operational domain models?

In operational systems, data distributions change over time due to code deployments, schema migrations, and dependency changes. Temporal splitting trains on data before time T and evaluates on data after time T, accurately simulating production deployment where models can only predict future unseen events.

What is benchmark decontamination, and how is it audited?

Benchmark decontamination ensures no test questions or solutions from evaluation suites appear in the training data. It is audited using n-gram containment checks (typically 8-word or 13-word contiguous sequences); any training document exceeding the overlap threshold is purged before training.

Related lessons

• Fine-Tuning & Domain-Specific Models · Decide Whether to TrainDiagnose the Gap Before Changing Weights12 min
• Fine-Tuning & Domain-Specific Models · Decide Whether to TrainSelect a Base Model for the Task13 min
• Fine-Tuning & Domain-Specific Models · Decide Whether to TrainSet a Baseline and Success Threshold12 min
• Fine-Tuning & Domain-Specific Models · Make Training Data TrustworthyTurn Real Tasks Into Training Examples14 min
• Fine-Tuning & Domain-Specific Models · Make Training Data TrustworthyFormat Conversations and Mask the Loss13 min