ML Evaluation Metrics Explained: Accuracy, Precision, Recall, F1 and ROC-AUC with Worked Examples

A model with "97% accuracy" that misses every fraud transaction. A classifier with "92% accuracy" that a doctor cannot use. These are not hypothetical — they are what happens when the metric does not match the problem, and it is one of the most common weaknesses in final-year project reports: a single accuracy number, quoted without context, defending a model it does not describe.

This guide explains the evaluation metrics you will actually use — accuracy, precision, recall, F1, ROC-AUC for classification; MAE and RMSE for regression — with worked examples you can adapt directly into your report. The examples mirror the tabular ML projects students build, like credit card fraud detection and disease prediction systems. Pair this with the overfitting guide (metrics tell you what is wrong; it tells you how to fix it) and the report-writing guide when you write it all up.

The confusion matrix: where every metric begins

Every classification metric is arithmetic on four counts. For a binary problem (say, fraud vs legitimate), compare predictions against truth:

	Predicted positive (fraud)	Predicted negative (legitimate)
Actually positive (fraud)	True Positives (TP)	False Negatives (FN) — missed fraud
Actually negative (legitimate)	False Positives (FP) — false alarms	True Negatives (TN)

Worked example — a fraud detector tested on 1,000 transactions, of which 50 are actually fraud:

• TP = 40 (caught 40 of the 50 frauds)
• FN = 10 (missed 10 frauds)
• FP = 30 (flagged 30 legitimate transactions)
• TN = 920

Keep these four numbers in front of you; everything below is derived from them.

Accuracy — and why it lies

Accuracy = (TP + TN) / (TP + TN + FP + FN) = (40 + 920) / 1000 = 0.96

96% accuracy sounds excellent. Now consider the laziest possible model: predict "legitimate" for every transaction. TP = 0, FN = 50, FP = 0, TN = 950. Accuracy = 950/1000 = 0.95. A model that catches zero fraud scores 95% — one point below your "good" model. This is the accuracy paradox: on imbalanced data, accuracy measures mostly how well you predict the majority class, which is the class nobody needed a model for.

Rule: if your classes are imbalanced (fraud, disease, defects — the interesting class is almost always the rare one), accuracy is the last metric you look at, not the first. Report it for completeness; decide by the metrics below.

Precision and recall: the pair that matters

Precision = TP / (TP + FP) = 40 / (40 + 30) = 0.571
Recall    = TP / (TP + FN) = 40 / (40 + 10) = 0.800

• Precision: of everything the model flagged, what fraction was actually fraud? 57.1%. Low precision means false alarms — investigators waste time, customers get insulted, doctors order unnecessary tests.
• Recall (sensitivity): of all actual fraud, what fraction did the model catch? 80%. Low recall means misses — fraud goes through, disease goes undiagnosed.

Precision and recall pull in opposite directions. Lower the decision threshold (flag more transactions) and recall rises while precision falls; raise it and the reverse happens. There is no universally right balance — it depends on the cost of each error type:

Project	Costly error	Optimise for
Fraud detection	Missed fraud (FN) costs money directly	Recall — accept false alarms
Disease screening	Missed disease (FN) costs health	Recall — false positives get follow-up tests
Spam filtering	False positive (FP) hides real email	Precision — missing a spam is harmless
Defect detection on a line	Depends: scrapping good parts (FP) vs shipping defects (FN)	State the tradeoff explicitly in your report

The sentence "we optimised for recall because a missed tumor costs more than an unnecessary follow-up" is exactly the kind of domain reasoning examiners want to hear. Write it in your report.

F1 score: the harmonic mean, and why harmonic

F1 = 2 × (Precision × Recall) / (Precision + Recall)
   = 2 × (0.571 × 0.800) / (0.571 + 0.800)
   = 2 × 0.4568 / 1.371 ≈ 0.667

The F1 score combines precision and recall into one number. It uses the harmonic mean rather than the ordinary average deliberately: the harmonic mean punishes extreme imbalance. A model with precision 1.0 and recall 0.1 averages to 0.55 arithmetically but scores F1 ≈ 0.18 — correctly signalling that a model finding 10% of fraud is poor despite never false-alarming. When you need one number for an imbalanced problem and cannot justify favouring precision or recall, F1 is the default.

Variants you will meet: macro-F1 averages the per-class F1 scores equally (treats rare and common classes the same — usually what you want to report); micro-F1 aggregates the raw TP/FP/FN first (dominated by the majority class — rarely what you want on imbalanced data); weighted-F1 averages per-class F1 weighted by class size. Always say which one you report. "F1 = 0.81" without the variant is an incomplete result.

ROC-AUC: threshold-independent quality

Precision, recall, and F1 all depend on the decision threshold (usually 0.5). ROC-AUC asks a threshold-free question: if I pick a random positive and a random negative example, what is the probability the model scores the positive higher? 0.5 is random guessing; 1.0 is perfect separation.

• Strengths: a single number summarising ranking quality across all thresholds; useful for comparing models.
• Weakness on imbalanced data: ROC-AUC can look flattering when negatives dominate, because the false-positive rate stays small even with many false alarms in absolute terms. On heavily imbalanced problems, the Precision-Recall AUC is the more honest curve — it focuses on the rare class's tradeoff directly.
• What to report: ROC-AUC for balanced problems; PR-AUC (average precision) for imbalanced ones; both if you want to be thorough. Name the choice in your report.

Multi-class: the same ideas, per class

For problems with 3+ classes (disease prediction across several conditions, waste categories), compute precision, recall, and F1 per class, then average with macro-F1. Always show the per-class table — it reveals what the average hides:

Class	Precision	Recall	F1	Support
Condition A	0.91	0.88	0.89	320
Condition B	0.74	0.69	0.71	180
Condition C	0.52	0.44	0.48	60
Macro average	0.72	0.67	0.69	560

The headline "F1 0.69" is honest; the per-class rows show Condition C is the weak point — probably the rarest class with the fewest examples. That observation is your "future work" section: targeted data collection for Condition C.

Regression metrics: MAE, RMSE, and R²

For projects predicting a number (house prices, energy output, marks prediction):

• MAE (Mean Absolute Error): the average of |predicted − actual|. In the original units — "off by ₹4,200 on average" — and robust to outliers.
• RMSE (Root Mean Squared Error): the square root of the average squared error. Same units, but squaring punishes large errors disproportionately. If being badly wrong occasionally is worse than being slightly wrong often, RMSE is your metric.
• R²: the fraction of variance explained, 0 to 1 (roughly). 0.85 means the model explains 85% of the target's variation. Intuitive, but gameable — adding junk features can inflate it, so report it alongside MAE/RMSE, never alone.

Worked comparison: predictions off by [2, 3, 2, 25] on four examples. MAE = (2+3+2+25)/4 = 8. RMSE = √((4+9+4+625)/4) = √(160.5) ≈ 12.7. The single large error barely moves MAE but dominates RMSE — which tells you the model has an outlier problem that MAE alone would hide. Report both; the gap between them is diagnostic.

Which metric for which project: the decision table

Situation	Report	Why
Balanced classes, errors cost the same	Accuracy + F1	Accuracy is meaningful here; F1 adds the tradeoff view
Imbalanced classes (fraud, disease, defects)	Per-class precision/recall, macro-F1, PR-AUC	Accuracy lies; the rare class is the point
Missing positives is costly (screening)	Recall, then F1	Say so explicitly and set the threshold accordingly
False alarms are costly (spam, auto-actions)	Precision, then F1	Same — threshold choice follows the cost
Comparing two models fairly	ROC-AUC or PR-AUC	Threshold-independent; add F1 at the operating threshold
Regression	MAE + RMSE (+ R²)	Units people understand, plus outlier sensitivity
Object detection	mAP@0.5 and mAP@0.5:0.95	Different machinery — see the YOLO guide's metrics section

Computing them: the five lines that prevent arithmetic errors

from sklearn.metrics import (accuracy_score, precision_score, recall_score,
                             f1_score, roc_auc_score, confusion_matrix,
                             classification_report)

y_true = [...]  # true labels, test set only
y_pred = [...]  # model predictions
y_scores = [...]  # predicted probabilities for the positive class

print(confusion_matrix(y_true, y_pred))
print(classification_report(y_true, y_pred))  # per-class P/R/F1 + macro avg
print("ROC-AUC:", roc_auc_score(y_true, y_scores))

classification_report is the single most useful output in student ML: per-class precision, recall, F1, support, and macro/weighted averages in one table — paste it into your report (as a formatted table, not a screenshot of a terminal).

Calibration: is "90% confident" right 90% of the time?

A model that outputs probabilities carries an extra obligation: the probabilities should mean something. A well-calibrated fraud model saying "90% probability of fraud" should be right about 9 times out of 10 on such cases. Many models are not — neural networks in particular tend toward overconfidence, outputting 0.99 for predictions that are right 80% of the time.

Why it matters for your project: if your system auto-blocks transactions above a 0.9 threshold, miscalibration decides how many legitimate users get blocked. Check calibration with a reliability diagram — bucket predictions by confidence (0.5–0.6, 0.6–0.7, …) and plot each bucket's actual accuracy. Points near the diagonal are calibrated; points below it are overconfident. If yours is badly off, simple fixes exist (temperature scaling on the validation set is the standard one-liner), but even reporting the diagram puts your evaluation ahead of most student reports. Mention it in one paragraph; examiners who know ML will notice.

Comparing two models: is the difference real?

"Model A: F1 0.81. Model B: F1 0.83. B is better." Is it? On a test set of 500 examples, a 2-point gap may be noise. Two honest ways to check:

• Cross-validation spreads. If A's folds give 0.81 ± 0.03 and B's give 0.83 ± 0.03, the intervals overlap heavily — you cannot claim B is better. If B gives 0.83 ± 0.01 across folds and A 0.78 ± 0.02, the claim is solid.
• McNemar's test for paired classifiers on the same test set: it looks specifically at the examples where the models disagree and asks whether B's wins significantly outnumber A's. Available in a few lines via statsmodels. You do not need deep statistics — you need the sentence "the difference was significant at p < 0.05 by McNemar's test on the held-out test set" instead of "B scored higher."

The practical rule: differences under ~2 points on a few hundred test examples are suggestive, not conclusive. Either gather more test data or present both models honestly.

Ranking metrics: when order matters more than labels

Some projects do not classify — they rank. A fraud system surfaces the 100 most suspicious transactions for review; a recommendation component lists the top 10. Here the question is not "is each prediction right?" but "are the true positives near the top?" Useful metrics:

• Precision@k: of the top k ranked items, what fraction are positive? "Precision@100 = 0.34" tells the review team that a third of their queue is real fraud — an operational number, not an abstract one.
• Cumulative gains / lift charts: plot what fraction of all positives you capture as you go deeper into the ranked list. If the top 10% of scores contain 70% of the fraud, the chart shows it in one glance — and it is an excellent report figure because non-technical examiners read it instantly.

If your project has any triage or prioritisation component, add one ranking metric alongside the classification ones. It connects the model to how it will actually be used.

A sample results write-up you can adapt

Below is a compact results section in the style this guide recommends — concrete numbers, named variants, stated tradeoffs. Adapt the structure, never the numbers:

The fraud detector was evaluated once on a held-out test set of 1,000 transactions (50 fraud, 950 legitimate), split by time — the last two weeks of data — so the test reflects future transactions, not a random sample of the past. The majority-class baseline scores F1 0.00 on the fraud class (it predicts no fraud) at 0.95 accuracy, confirming that accuracy alone is uninformative here.

At the operating threshold of 0.5, the model achieves fraud-class precision 0.57 and recall 0.80 (macro-F1 0.72, PR-AUC 0.74). The threshold was chosen to favour recall: a missed fraud costs the bank the transaction value (average ₹18,400 in our data), while a false alarm costs one manual review (~4 minutes of analyst time). Lowering the threshold to 0.3 raises recall to 0.88 at precision 0.41; the 0.5 operating point was preferred because review-team capacity caps false alarms at ~40 per day.

Error analysis on the 10 missed frauds shows 6 were first-time transaction patterns with no historical features — a data limitation, not a model failure — and 4 were amounts just below the review threshold, suggesting threshold gaming. These form the future-work items: behavioural features for new accounts, and adaptive thresholds.

Every sentence in that write-up answers a question an examiner would ask: how was it tested, what is the baseline, why this threshold, what do the errors mean. Write yours the same way.

Common examiner questions on metrics (and strong answers)

• "Why didn't you just use accuracy?" — "Because 5% of our data is fraud, a model predicting 'legitimate' always scores 95% accuracy while catching nothing. We report accuracy for completeness but decide on fraud-class recall and macro-F1."
• "Your precision is only 0.57 — isn't the model bad?" — "Precision 0.57 means 4 in 10 alarms are false. That is the price of recall 0.80 given our data, and it fits the review team's capacity of 40 manual checks per day. The tradeoff was chosen, not stumbled into."
• "Why macro-F1 and not weighted?" — "Weighted F1 is dominated by the majority class. Macro-F1 treats the fraud class as equally important, which matches the project's goal."
• "How do you know the test set wasn't leaked?" — "Split by time: the test set is the final two weeks, never used in training or validation, and the split script with its seed is in the appendix."
• "Would the metrics hold on new data?" — "The 5-fold cross-validation spread was ±0.03 around the mean, which bounds the expected variation. True drift — fraud patterns changing — needs the monitoring described in future work."

Prepare these before the viva, not during it. Each answer points at an artifact (table, script, figure) in the report — metrics you can point to are metrics you own.

Explaining metrics to non-technical examiners

Your external examiner may not know ML. Prepare a 60-second plain-language version of your results:

• "The system catches 8 out of 10 frauds" (recall 0.80) — not "recall is 0.8."
• "When it raises an alarm, it is right about 6 times out of 10" (precision 0.57) — and immediately follow with why that tradeoff is acceptable: "a missed fraud costs the bank directly, while a false alarm costs one manual review."
• "We tested on 1,000 transactions the model had never seen" — the held-out test set, in plain words.

Then keep the full confusion matrix, per-class table, and ROC/PR curves in the report for the technical reader. The two-layer presentation — plain summary up front, rigorous detail behind it — is what makes a results chapter persuasive to a mixed panel. Rehearse the 60-second version out loud; metrics you cannot explain simply are metrics you do not fully understand yet.

Metric gotchas: the checklist before you quote a number

Run through this list before any metric enters your report or slides:

• Computed on the test set, which was used exactly once — not on validation, not on training.
• Every variant named: macro/micro/weighted F1, ROC-AUC vs PR-AUC, not bare "F1" or "AUC".
• Baseline included: majority-class or simple-model scores alongside yours.
• Class distribution of the test set stated next to the metrics.
• Threshold stated wherever precision/recall/F1 appear (they are threshold-dependent; ROC-AUC is not).
• No metric was selected after seeing which one looked highest — decide the primary metric from the problem's error costs before training, and say so.
• Cross-validation spread reported if the test set is small (under ~1,000 examples), so a lucky split cannot masquerade as a result.

That last point deserves emphasis: choosing your headline metric after trying several is a quiet form of overfitting — overfitting to the test set through metric selection. Decide up front, document the decision, and let the numbers fall where they do. Honest metrics that you can defend beat flattering metrics that collapse under one question.

How to present metrics in your report

Examiners read dozens of reports; make yours scannable and honest:

1. State the test setup first: dataset size, class distribution, split method. A metric without this context is uninterpretable.
2. Lead with the confusion matrix (or per-class table for multi-class), then the derived metrics. Readers should be able to verify your arithmetic.
3. Name every variant: "macro-F1", "PR-AUC" — never bare "F1" or "AUC".
4. Show the baseline: a majority-class classifier's scores, or a simple model's. Your model's numbers mean nothing without the "doing nothing scores X" anchor.
5. Discuss errors, don't hide them: two or three misclassified examples with why they were hard (blurry image, borderline values) signal understanding. The testing and debugging guide covers building this failure analysis.
6. Never tune on the test set. The test set is measured once, at the end. Every threshold tweak against it leaks optimism into your headline numbers.

Metrics are not decoration for the results chapter — they are the argument your project stands on. A report that says "recall 0.80 on the fraud class because missed fraud costs more than false alarms, macro-F1 0.72, evaluated once on a held-out test set of 1,000 transactions" will survive questioning that "accuracy 97%" never could.