Explainable AI for Student ML Projects: SHAP, LIME and Grad-CAM

Explainable AI for Student ML Projects: SHAP, LIME and Grad-CAM

Summary: "The model said so" is not an explanation. When a mentor, examiner or user asks why your model predicted what it did, you need an answer grounded in the model's actual behaviour. This guide explains the three explanation methods students actually use — SHAP for tabular models, LIME for text and image classifiers, and Grad-CAM for convolutional networks — how each one works conceptually, how to run them with modest Python code, how to read their outputs honestly, and the mistakes that make explanations misleading instead of useful.

The problem: accurate but unexplainable

Student ML projects have a predictable arc: you train a model, the accuracy looks good, and then someone asks the question you dread — why did it classify this particular input that way? Typical answers fail in predictable ways:

• "The neural network learned the pattern" — true of every neural network ever, explains nothing.
• Pointing at the accuracy number — accuracy describes the model on average, not this prediction.
• Showing a confusion matrix — again, aggregate behaviour, not this decision.

There are also practical reasons to want explanations beyond satisfying an examiner. When your crop-disease classifier keeps misclassifying healthy leaves, an explanation can show you it was looking at the background soil, not the leaf — which tells you your dataset has a background bias, not that you need a bigger model. Explanations are a debugging tool first and a presentation tool second.

Explainable AI (XAI) is the family of methods that attribute a prediction back to the input features. Three of them cover nearly every student use case.

How the three methods work (conceptually)

SHAP: fair credit assignment for tabular models

SHAP (SHapley Additive exPlanations) borrows an idea from game theory: if several players cooperated to win a game, how do you fairly divide the prize? Replace players with input features and the prize with the prediction, and you get per-feature contribution scores.

For a loan-default model, SHAP might say: base prediction 0.30, applicant income pushed it down by 0.12, missed payments pushed it up by 0.25, final prediction 0.43. The contributions add up to the difference between the average prediction and this one — that additivity is what makes SHAP plots readable.

SHAP works best with tabular data and tree models (random forests, XGBoost, LightGBM have a fast exact algorithm, TreeSHAP). It is slow on large neural networks, so don't force it there.

LIME: a simple local stand-in

LIME (Local Interpretable Model-agnostic Explanations) takes a different approach: it doesn't try to explain the whole model. It zooms in on one prediction, generates slightly perturbed versions of that input, sees how the model's output changes, and fits a tiny interpretable model (like a linear regression) on those samples.

For a text sentiment classifier, LIME might highlight that the words "not" and "disappointing" pushed the review toward negative. For an image classifier, it highlights superpixel regions that mattered. The key word is local: the explanation is valid near this input only, not as a global rule about the model.

Grad-CAM: where the CNN was looking

Grad-CAM (Gradient-weighted Class Activation Mapping) is built for convolutional networks. It uses the gradients of the target class flowing into the final convolutional layer to produce a heatmap over the input image: bright regions are where the network "looked" to make its decision.

This is the method behind those compelling demo images where the heatmap glows over the tumour, the defect, or the diseased leaf region. It needs no model retraining and works with any CNN — which is why it dominates student computer-vision projects.

Which to use when:

Your project	Method	Why
Tabular data (CSV features, XGBoost/RF)	SHAP	Exact, fast, additive — the gold standard here
Text classifier (sentiment, spam)	LIME	Highlights influential words per prediction
Image classifier (CNN, transfer learning)	Grad-CAM	Shows the regions the CNN used
Any model, quick sanity check	LIME	Model-agnostic, works on anything

Practical 1: SHAP on a tree model

The shap library with TreeSHAP on a trained XGBoost or random-forest model runs in seconds on student-size datasets:

import shap

# model: your trained tree model, X_test: your feature dataframe
explainer = shap.TreeExplainer(model)
shap_values = explainer.shap_values(X_test)

# summary: which features matter overall (save the figure for your report)
shap.summary_plot(shap_values, X_test, show=False)

# waterfall: why THIS row got THIS prediction
shap.plots.waterfall(shap.Explanation(
    values=shap_values[0],
    base_values=explainer.expected_value,
    data=X_test.iloc[0]))

Two plots, two jobs. The summary plot ranks features by overall importance — good for the report's methodology section. The waterfall plot decomposes a single prediction into per-feature pushes — good for the demo ("here is why this applicant was flagged").

Reading rule: a feature with a large SHAP value influenced this prediction strongly; it does not mean the feature causes the outcome in the real world. Correlation, not causation — write that sentence in your report and mean it.

Practical 2: LIME on a text classifier

from lime.lime_text import LimeTextExplainer

# predict_fn: takes a list of strings, returns class probabilities
explainer = LimeTextExplainer(class_names=["negative", "positive"])
exp = explainer.explain_instance(review_text, predict_fn, num_features=8)

# top contributing words for this prediction
print(exp.as_list())

LIME perturbs the text by randomly dropping words and watches the prediction move. as_list() gives you word–weight pairs: positive weights pushed toward the predicted class, negative weights pushed away. In your demo, render these as highlighted words — it is immediately legible to any audience.

Honesty note: LIME involves random sampling, so running it twice on the same input can give slightly different word rankings. Set a random seed, and never present one LIME run as the definitive explanation — it's a sampled approximation.

Practical 3: Grad-CAM on your CNN

If you used transfer learning (a common student pattern), Grad-CAM needs only your trained model. The widely used recipe with a library like pytorch-grad-cam:

from pytorch_grad_cam import GradCAM
from pytorch_grad_cam.utils.image import show_cam_on_image

# target_layer: the last convolutional layer of your model
cam = GradCAM(model=model, target_layers=[target_layer])
grayscale = cam(input_tensor=input_batch, targets=[ClassifierOutputTarget(class_idx)])
heatmap = show_cam_on_image(rgb_image, grayscale[0])

Overlay the heatmap on the original image and ask the hard question: is the hot region the object, or the background? Common student findings:

• The pneumonia classifier's heatmap glows on the image corner — it learned the hospital's watermark position, not lung pathology.
• The crop-disease heatmap glows on soil — the dataset's diseased photos were taken in different fields than the healthy ones.
• The defect detector glows on the conveyor belt texture — lighting differs between the two classes.

Each of these is a dataset problem that no amount of hyperparameter tuning fixes. This is Grad-CAM earning its place: it converts a mysterious accuracy plateau into an actionable data-collection task.

Mistakes that make explanations misleading

1. Explaining the wrong model. You tuned three models and explain the one with the prettiest heatmap instead of the one you actually deployed. Explain the final model, on final data.
2. Cherry-picking examples. Showing only the five predictions where the explanation looks sensible. Show a failure case too — examiners trust you more, and it usually reveals something real.
3. Treating explanations as proof of correctness. A beautiful heatmap over the right region does not prove the model is right for the right reasons on new data. It is evidence, not a certificate.
4. Ignoring the base rate. SHAP values are relative to the model's average prediction. A feature "pushing toward fraud" by +0.2 means little if the base rate is 0.01 — always report the base value alongside.
5. Running SHAP on a 10-million-row dataset naively. KernelSHAP is slow; use TreeSHAP for tree models and explain a sampled subset (a few hundred representative rows) rather than everything.
6. Leaking the explanation into training. Using explanation-derived feature selection on the same data you evaluate on is a form of leakage — do selection on the training split only.

FAQ

Do I need explainability if my accuracy is already high?
High accuracy on your test set doesn't tell you what the model learned. Explanations catch shortcut learning (backgrounds, watermarks, timestamps) that accuracy hides — and that an examiner probing your demo will find.

Which method should I put in my report if I only have time for one?
Match it to your data type: SHAP for tabular, Grad-CAM for CNN images, LIME for text. One well-executed method beats three half-understood ones.

Are these methods accepted in academic project reports?
Yes — SHAP, LIME and Grad-CAM are all published, peer-reviewed methods with thousands of citations. Cite the original papers, state the library version you used, and report your settings (sample sizes, seeds).

Can explanations be gamed or fooled?
Yes, and you should know this before your viva: researchers have shown explanations can be manipulated to hide biased behaviour. Mentioning this limitation yourself shows more understanding than pretending explanations are infallible.

My model is a scikit-learn logistic regression — do I need SHAP?
Probably not. Linear models are already interpretable: the coefficients are the explanation (after proper feature scaling). XAI methods earn their keep on non-linear models — trees, forests, neural networks.

Limitations

• These are post-hoc methods: they approximate or probe a trained model rather than making the model itself interpretable. They can be wrong about the model's true reasoning.
• Explanations describe correlation between inputs and outputs, never real-world causation.
• Compute cost is real: SHAP on large models and LIME's sampling loop can take minutes per explanation — plan this into your demo, don't compute live on stage.
• This guide covers the standard three methods. Counterfactual explanations ("what minimal change would flip this prediction") and concept-based methods are valuable but beyond student-project scope.

---

Related reading: pair this with our ML evaluation metrics guide so your numbers are sound before you explain them, and fixing overfitting for when explanations reveal your model memorised the training data. CNN users should also see transfer learning for image projects.