In this guide
Test Set Leakage: The Silent Mistake That Inflates Your ML Accuracy
You train a classifier on a nice tabular dataset. Your test accuracy comes out at 96%. You feel good. Then your guide asks you to test it on a fresh batch of data — records the model has truly never seen — and the accuracy collapses to the mid-60s. Nothing is wrong with your model architecture. Something leaked.
Test set leakage (often just called data leakage) is information from outside the training data reaching the training process — information the model will not have at prediction time. The test set is supposed to simulate the future. Leakage lets the model peek at the future. The score you get is then a measurement of the peek, not of the model.
This guide covers the five forms leakage takes in student projects, how to find it in notebooks you have already written, and the workflow habits that keep your results honest.
What leakage actually means
A machine learning model's job is to generalise: to make correct predictions on data it has not seen. The test set exists to simulate that unseen data. Any step where information derived from the test set (or from records the model should not have seen) influences what the model learns is leakage.
Leakage is not the same as overfitting. Overfitting is a model memorising training noise; the training process itself was legitimate. Leakage is a corrupted evaluation: the training process had access to information it should not have. An overfit model's test score drops a bit; a leaky model's test score is simply wrong as an estimate.
The confusing part: leakage almost always inflates scores. If your model performs suspiciously well — 95%+ on a messy real-world dataset where published baselines sit much lower — leakage is the first thing to check, not the last.
Leak 1: preprocessing fit on the full dataset
This is the most common leak in student code, and it hides in three innocent lines:
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X) # fit on ALL the data
X_train, X_test, y_train, y_test = train_test_split(X_scaled, y)
The scaler learns the mean and standard deviation from every row, including the test rows. The training data then carries information about the test set's distribution. The correct order is split first, fit second:
X_train, X_test, y_train, y_test = train_test_split(X, y)
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train) # fit on train only
X_test = scaler.transform(X_test) # test only gets transformed
The same applies to imputation (filling missing values with the column mean), one-hot encoding categories (categories seen in the test set), and text vectorisers (vocabulary built from test documents). Any transformer that learns something from data must learn it from the training split only.
The robust habit is to wrap everything in a scikit-learn Pipeline, where fitting happens on the training data alone by construction:
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
pipe = Pipeline([
("scaler", StandardScaler()),
("model", LogisticRegression()),
])
pipe.fit(X_train, y_train) # scaler fit on train only, automatically
print(pipe.score(X_test, y_test))
A Pipeline does not remove all leaks — feature choices you made by staring at the whole dataset are still leaks — but it eliminates the entire class of preprocessing-order mistakes.
Leak 2: feature selection and statistics computed on all the data
A subtler version: you run SelectKBest, recursive feature elimination, or even just "keep the columns that correlate with the label" on the full dataframe before splitting. The selected features are now chosen partly because of their relationship with the test rows. The model trains on features that were pre-screened against data it should not know.
Related leaks in the same family:
- Target encoding (replacing a category with its mean label value) computed over the full dataset, including test rows.
- Dropping "unhelpful" columns because they looked useless across the whole dataset.
- Balancing classes (oversampling the minority class) before the split, so duplicated minority rows land in both train and test.
The rule is the same: anything that uses the labels to make a decision must make that decision from the training split alone. Feature selection belongs inside the Pipeline or inside each cross-validation fold, not in a preprocessing cell above the split.
Leak 3: duplicates and near-duplicates across the split
If the same record (or near-identical records) appears in both the training and test sets, the model is effectively being tested on data it trained on. This happens more often than students expect:
- The dataset genuinely contains duplicate rows (very common in scraped or merged student datasets).
- Augmented copies: you create rotated/flipped image variants or paraphrased text samples and split after augmentation, so a test image is a rotated twin of a training image.
- Entity-level duplicates: medical datasets with multiple rows per patient, where patient 42 appears in both splits. Splitting rows is not splitting patients.
- Time-series windows: overlapping sliding windows where a test window shares most of its samples with a training window.
Check before you split: look for duplicate rows, and if your data has a natural grouping (patient, device, user, session), split by group, not by row. scikit-learn's GroupKFold exists exactly for this — use it instead of plain KFold whenever records cluster under a shared entity.
Leak 4: features that use the future
Some features quietly encode the answer. Classic examples:
- Predicting loan default with a feature like "days since last repayment" computed from the full repayment history — including repayments that happened after the prediction date.
- Predicting equipment failure using a sensor log that was trimmed using the failure timestamp.
- A "user churned" label derived from behaviour in the same window as the features.
The test: for every feature, ask "would I know this value at the moment I must make the prediction?" If a feature could only be computed after the outcome is known, it is leakage. In student projects this usually appears when the dataset was assembled carelessly — columns that look informative are informative precisely because they were derived from the answer.
Leak 5: shuffling time-ordered data
When data has a timestamp, random shuffling is itself a leak: the model trains on the future and is tested on the past. Real deployments predict the future from the past, never the reverse. For time-ordered data, split by time — the earliest records train, the latest test — and validate with TimeSeriesSplit, which respects the ordering inside cross-validation.
Student projects hit this with stock prices, sensor streams, and any "predict next month" dataset that someone casually shuffled with train_test_split.
How to audit a project you have already built
Run through this checklist on your existing notebooks:
- Find your train_test_split call. Is every fit_transform, feature-selection step, and target-encoding step above or below it? Anything above it that touches y or the full X is suspect.
- Search for duplicate rows and duplicate entity IDs across your splits.
- List your features and apply the "would I know this at prediction time?" test to each.
- Check whether your data is time-ordered. If it is, re-split by time and compare scores — a big drop is diagnostic.
- Re-run your pipeline with a GroupKFold or TimeSeriesSplit if either applies, and compare against your original score.
If your score drops after fixing leaks, that is not your model getting worse — that is your measurement becoming honest. An honest 72% you can defend in a viva is worth more than a 96% you cannot explain.
Frequently asked questions
Is leakage the same as overfitting?
No. Overfitting means the model memorised training noise; the evaluation was fair but the model is too complex. Leakage means the evaluation itself was corrupted because training saw information it should not have. You can have a simple, well-regularised model that still reports inflated scores because of leakage.
Doesn't cross-validation protect me?
Only if the leakage is not baked in before the folds. If you scale the full dataset and then cross-validate the scaled data, every fold is contaminated. Preprocessing and feature selection must happen inside each fold — which is exactly what Pipelines do when you pass them to cross_val_score.
My college gave me a fixed train/test split. Can I still leak?
Yes, easily. The fixed test set only protects you if you never let it influence training. Fitting your scaler on the combined data, tuning features against the test score repeatedly, or "just checking" the test set while iterating all count. Treat the provided test set as sealed: use it once, at the end.
Is leakage worse on small datasets?
Generally yes. With a few hundred rows, the test set's contribution to statistics like column means is proportionally large, duplicates are more likely to straddle the split, and a leaky feature has an outsized effect. Small student datasets need the most discipline, not the least.
I fixed everything and my accuracy dropped 20 points. What do I do?
First, make sure the drop is really from fixing leaks and not from a bug you introduced — re-check your pipeline order. Then report the honest number and explain the drop in your report: examiners respect a student who can say "the first number was inflated by leakage from scaling before the split; the corrected evaluation is X" far more than one who defends an indefensible 97%.
Limitations
- This guide focuses on supervised learning with tabular data; image, text and audio projects have their own leakage modes (augmented twins, same-speaker recordings in both splits) that follow the same principles but need domain-specific checks.
- The numeric examples in this guide are illustrative of the mechanism, not measured results from a specific dataset — the size of the inflation depends entirely on your data.
- No checklist replaces understanding your own dataset: the "would I know this at prediction time?" test requires you to know how each feature was constructed.
Suitable for B.E./B.Tech final-year projects in Computer Science, IT, AI/ML and Data Science where models are trained and evaluated on real datasets.