In this guide
Your fraud detector sees 10,000 transactions. Fifty are fraud. A model that predicts "legit" for everything scores 99.5% accuracy — and catches zero fraudsters. Accuracy lied, and it will keep lying on every imbalanced dataset you ever train on: defect inspection, churn prediction, disease screening, spam filtering.
Class imbalance is not a data problem first. It is a measurement problem first, then a cost problem, then a data problem. This guide takes the fixes in the order you should actually try them: fix the metric, move the threshold, weight the classes, and only then touch the data with SMOTE or undersampling. Each fix gets its honest trade-offs, because every one of them can backfire.
Start by fixing the metric, not the model
Accuracy counts every mistake equally, which is exactly wrong when your classes are 200:1. A confusion matrix splits the errors into four:
| Predicted fraud | Predicted legit | |
|---|---|---|
| Actually fraud | True positives (caught) | False negatives (missed fraud) |
| Actually legit | False positives (false alarm) | True negatives |
Precision = TP/(TP+FP): of everything flagged, how much was real? Recall = TP/(TP+FN): of all real fraud, how much did you catch? A single number: F1 is the harmonic mean of the two; PR-AUC (area under the precision-recall curve) summarises performance across thresholds and is far more honest than ROC-AUC on skewed data.
Pick the metric from the cost of mistakes. In fraud or disease screening, a missed case costs far more than a false alarm — optimise recall (or F-beta with beta above 1, which weights recall). In spam filtering or ad-click prediction, false alarms annoy users — weight precision (beta below 1). Most students skip this step and spend weeks resampling data to fix what was actually a metric choice.
Fix 1: move the decision threshold (free, do this first)
Most classifiers output probabilities; the 0.5 decision threshold is just a default. Plot precision against recall at different thresholds on your validation set and pick the point that matches your cost trade-off. In fraud detection, moving the threshold from 0.5 to 0.3 might double recall while precision dips from 0.8 to 0.5 — often exactly the trade you want.
probs = model.predict_proba(X_val)[:, 1]
preds = (probs >= 0.3).astype(int) # lower threshold, higher recall
print(classification_report(y_val, preds))
No retraining, no data tricks, takes five minutes. If threshold tuning already gives you the recall you need, stop here — the fancier fixes add complexity without paying rent.
Fix 2: class weights (cheap, try second)
Class weights tell the loss function that minority-class mistakes hurt more. In scikit-learn, one parameter does it:
from sklearn.ensemble import RandomForestClassifier
clf = RandomForestClassifier(class_weight="balanced") # weight ~ 1/class frequency
Intuition: if fraud is 1% of the data, each fraud example counts roughly 100 times in the loss. The decision boundary shifts toward catching the minority. It works with almost any estimator (logistic regression, SVM, gradient boosting — XGBoost and LightGBM expose equivalents such as scale_pos_weight), adds no training time worth mentioning, and cannot leak data because nothing about the data changes.
Trade-offs: weighting makes probability calibration worse (a 0.7 output stops meaning "70% likely"), so re-threshold after weighting instead of trusting 0.5. On very tiny minorities (a few dozen examples), weights just amplify noise — the model memorises a handful of fraud cases instead of learning the pattern.
Fix 3: SMOTE and oversampling (powerful, easy to misuse)
Random oversampling duplicates minority rows until the classes balance. The model then sees the same fraud cases dozens of times — and overfits to them. SMOTE (Synthetic Minority Over-sampling Technique) is smarter: it picks a minority example, finds its k nearest minority neighbours, and synthesises new points along the lines between them.
from imblearn.over_sampling import SMOTE
sm = SMOTE(random_state=42)
X_train_res, y_train_res = sm.fit_resample(X_train, y_train) # train split only
The capitalised rule: resample the training fold only, inside your cross-validation, never the whole dataset before splitting. SMOTE before the split bleeds synthetic neighbours of validation points into training — your validation scores look wonderful and your production scores do not. Use imbalanced-learn's Pipeline so every fold resamples correctly.
SMOTE helps most when the minority class has real structure but too few samples to reveal it, and the classes are reasonably separable. It fails when the classes genuinely overlap (synthesising points in the overlap region just adds confident-looking noise), on tiny minorities (neighbours of noise are noise), and on high-dimensional sparse data like text. Variants like Borderline-SMOTE and ADASYN focus synthesis near the decision boundary — worth trying, not magic.
Fix 4: undersampling the majority (fast, wasteful)
Random undersampling deletes majority examples until the classes balance. Training gets dramatically faster and cheaper — a real consideration with millions of rows. Cleaning variants (Tomek links, edited nearest neighbours) remove only majority examples near the boundary, keeping the bulk of the data.
The cost is obvious: you throw away data. When you have millions of majority rows, nobody misses them. When your whole dataset is 2,000 rows, deleting 1,900 is self-sabotage. A gentler version is ensemble undersampling: train several models on different majority subsets and average them — you use all the data without the imbalance. And you can combine: mild undersampling of the majority plus SMOTE of the minority often beats either alone.
Which fix first: the decision table
| Situation | Start with |
|---|---|
| Any imbalanced problem | Threshold tuning with a PR-based metric |
| Model supports weights, minority in the hundreds | class_weight="balanced" |
| Minority below ~100 examples | Class weights; avoid SMOTE on noise-scale minorities; consider collecting more data |
| Millions of rows, training too slow | Undersampling or cleaning variants |
| Separable classes, moderate minority | SMOTE inside a pipeline, validated properly |
| Classes heavily overlapping | Weights plus threshold; resampling rarely saves overlap |
One fix students overlook entirely: change the problem. Can you frame it as anomaly detection (one-class) instead of classification? Can you collect more minority examples? Ten more real fraud cases beat any synthetic trick.
Common mistakes
- Resampling before the train/test split — data leakage; the single most common reason a class-imbalance "fix" looks amazing in the notebook and fails in the demo.
- Optimising accuracy — on 99:1 data, accuracy rewards the do-nothing model.
- SMOTE on the test set — synthetic test points measure nothing real.
- Judging with the same skewed metric after changing the data distribution.
- Ignoring the business cost — a technically better F1 that flags ten times more false alarms can be the worse model in practice.
- Resampling when the problem is overlap — if a human cannot tell the classes apart either, no sampling trick invents signal.
FAQ
Is class imbalance always a problem?
No. If the minority class is large in absolute terms (say 5,000 fraud cases among 200,000 legit) and the model already recalls well on it, steep ratios are fine. Imbalance hurts when the absolute minority count is small or the metric rewards ignoring it.
Which is better: SMOTE or class weights?
Neither universally. Class weights are simpler, faster and leak-proof — try them first. SMOTE wins when the minority class has learnable structure that more (synthetic) examples reveal. Published comparisons are mixed; your validation set decides.
Does SMOTE work for image or text data?
Poorly. Interpolating raw pixels or token sequences produces garbage neighbours. For images, use domain augmentation (rotations, crops); for text, back-translation or paraphrase augmentation. SMOTE shines on tabular and structured features.
Should I balance to exactly 50:50?
Rarely optimal. Try ratios like 1:2 or 1:3 minority to majority, and tune the threshold afterward. Full 50:50 often over-corrects and tanks precision.
My model has great validation F1 but fails in production. What did I miss?
Check for leakage first (resampling outside the fold), then distribution shift: fraud patterns drift, so a model validated on last month's data can fail this month. Monitor live precision and recall, not just training metrics.
Can I combine techniques?
Yes — mild undersampling of the majority plus SMOTE of the minority, with class weights on top, is a legitimate stack. Validate each addition; stacking fixes without validation is how complexity grows without performance.
Limitations
- Nothing here creates information: if the minority class carries no learnable signal, no fix invents it — collect better features or more real examples.
- SMOTE variants are research tools; cite the original Chawla et al. (2002) paper in your report rather than claiming you invented the method.
- Probability calibration degrades under weighting and resampling — if your application needs honest probabilities (risk scoring), recalibrate with Platt scaling or isotonic regression afterward.
- Thresholds chosen on one month's data decay as class distributions drift; production systems re-tune periodically.