CNN Transfer Learning: A Student's Guide

You have 800 photos of plant leaves and a deadline. Training a convolutional neural network from scratch on 800 images will give you a model that memorises the training set and embarrassment at the demo. Transfer learning is the way out: take a network already trained on millions of images (ImageNet), keep what it learned about edges, textures, shapes, and patterns, and retrain only the final layers on your classes. It is the single highest-leverage technique in student image classification, and it powers real builds like dog breed classification with transfer learning, plant disease detection with transfer learning, and bird species classification with EfficientNet.

This guide explains what transfer learning actually does inside the network, how to choose between ResNet and MobileNet, the exact PyTorch workflow (freeze, replace the head, fine-tune), how much data you really need per class, and the errors that waste the most student time.

What is transferred, exactly

A CNN builds features in layers: early layers detect edges and colour blobs, middle layers detect textures and simple shapes, late layers detect object parts and class-specific patterns. The crucial insight is that early and middle features are generic — an edge detector learned from ImageNet's 14 million images works just as well on plant leaves, dog fur, or bird feathers. Only the late layers are specific to the original 1,000 ImageNet classes.

Transfer learning exploits this hierarchy:

Load a pretrained backbone (e.g. ResNet50 trained on ImageNet).
Freeze the early layers — their weights stay fixed; they are already excellent feature extractors.
Replace the final classification layer with a new one sized for your classes (randomly initialised).
Train on your data. Initially only the new head learns; optionally, later unfreeze some late layers and fine-tune them gently.

Because the backbone arrives already competent, your few hundred images per class only need to teach the head — a far smaller learning problem than training millions of parameters from nothing. This is also why transfer learning is one of the strongest defences against overfitting on small datasets, as covered in the overfitting guide.

Choosing the backbone: ResNet vs MobileNet vs EfficientNet

Backbone	Parameters	ImageNet top-1 (approx.)	Strengths	Pick it when
MobileNetV2 / V3	~3–5M	~0.71–0.75	Tiny, fast; runs on phones and Raspberry Pi	Edge deployment, or the default student choice for simple problems
ResNet50	~25M	~0.76	The workhorse: well-understood, debuggable, abundant examples	Default for laptop/server demos; fine-grained classes
ResNet18	~11M	~0.70	Lighter ResNet; trains fast	Quick experiments, weaker hardware
EfficientNet-B0…B4	5–19M	~0.77–0.83	Strong accuracy-per-parameter	When you want the strongest small model and can handle slightly trickier training

Decision rules:

Demo on a laptop or server? ResNet50. It is the most documented backbone on earth; every error you hit has a Stack Overflow answer.
Demo on a Raspberry Pi or a phone? MobileNetV2/V3. A ResNet50 can run on a Pi, but slowly — prove your frame rate early.
Fine-grained classes (dog breeds, bird species — visually similar classes)? Favour the larger backbone (ResNet50 or EfficientNet) and more unfreezing, because the distinguishing features live in the late layers.
Distinct classes (healthy vs diseased leaf, waste categories)? MobileNetV2 is usually plenty.

Design target: on 300–800 images per class with distinct classes, a frozen-backbone ResNet50 or MobileNetV2 commonly reaches validation accuracy in the 0.85–0.95 range. Fine-grained problems (50+ similar breeds) need more data and more fine-tuning to get there. These are planning figures — verify on your own validation split.

How much data per class?

The honest planning numbers for transfer learning:

~100–200 images per class: workable for 2–5 visually distinct classes with a frozen backbone. Expect to lean on augmentation.
~300–800 per class: the comfortable zone for most student projects.
~1,000+ per class: fine-grained problems (breeds, species, disease stages) start becoming reliable here.
Under ~50 per class: few-shot territory — transfer learning alone struggles; you need heavy augmentation, and you should scope the project down (fewer classes) rather than hope.

These assume real variation (lighting, angle, background), not 500 frames from one video. And they assume clean labels — with 200 images per class, ten mislabelled ones are 5% label noise, which the model will feel.

The PyTorch workflow, step by step

PyTorch with torchvision is the standard student stack for transfer learning. The full pattern:

Step 1: Load the pretrained backbone

import torch
import torch.nn as nn
from torchvision import models

# ResNet50 with ImageNet weights (torchvision >= 0.13 API)
weights = models.ResNet50_Weights.IMAGENET1K_V2
model = models.resnet50(weights=weights)

Use the newest weight version available (IMAGENET1K_V2 beats V1 measurably). The weights download once and cache locally.

Step 2: Freeze the backbone, replace the head

# Freeze everything: no gradients for the backbone
for param in model.parameters():
    param.requires_grad = False

# Replace the final fully-connected layer for YOUR class count
num_classes = 4  # e.g. 4 plant-disease categories
in_features = model.fc.in_features
model.fc = nn.Linear(in_features, num_classes)
# model.fc parameters have requires_grad=True by default: only the head trains

For MobileNetV2, the head is model.classifier[1] (a Linear layer) rather than model.fc — check the architecture: print(model) once and look at the final layer's name. Getting the layer name wrong is a common silent error (you train nothing, or everything).

Step 3: Train the head

optimizer = torch.optim.Adam(model.fc.parameters(), lr=1e-3)
criterion = nn.CrossEntropyLoss()

model.train()
for epoch in range(15):
    for images, labels in train_loader:
        optimizer.zero_grad()
        outputs = model(images)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

Only the head's parameters are passed to the optimizer — the frozen backbone cannot move even if you make a mistake elsewhere. Train 10–20 epochs; the head converges fast because the features are already good.

Step 4: Fine-tune (unfreeze late layers, gently)

If validation accuracy plateaus below your target, unfreeze the last block or two and continue training with a much smaller learning rate:

# Unfreeze layer4 (the last ResNet block) for fine-tuning
for param in model.layer4.parameters():
    param.requires_grad = True

optimizer = torch.optim.Adam(
    [
        {"params": model.layer4.parameters(), "lr": 1e-5},  # gentle: pretrained
        {"params": model.fc.parameters(), "lr": 1e-4},      # head can move more
    ]
)

The learning-rate discipline is the whole game: 1e-5 to 1e-4 for pretrained layers, up to 1e-3 for the new head. A large learning rate on pretrained layers destroys the transferred features in a few batches — the classic "fine-tuning made it worse" story. If fine-tuning hurts, the rate was too high or the layers should have stayed frozen.

Step 5: The data pipeline

from torchvision import transforms, datasets
from torch.utils.data import DataLoader, random_split

train_tf = transforms.Compose([
    transforms.Resize((224, 224)),
    transforms.RandomHorizontalFlip(),
    transforms.RandomRotation(15),
    transforms.ColorJitter(brightness=0.2, contrast=0.2),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406],
                         std=[0.229, 0.224, 0.225]),  # ImageNet stats
])

# No augmentation on validation: resize + normalize only
val_tf = transforms.Compose([
    transforms.Resize((224, 224)),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406],
                         std=[0.229, 0.224, 0.225]),
])

full = datasets.ImageFolder("data/leaves", transform=train_tf)
n_val = int(0.2 * len(full))
train_set, val_set = random_split(full, [len(full) - n_val, n_val])
# NOTE: give val_set the val_tf transform (wrap or re-create the dataset)

Two details students get wrong: normalization must use ImageNet's mean/std (the backbone was trained on normalized inputs; feeding it raw 0–1 tensors degrades everything), and input size must match what the backbone expects (224×224 for ResNet/MobileNet; EfficientNet variants use larger sizes — check the docs).

Freezing strategy: how much to unfreeze

Strategy	What trains	When
Head only (backbone frozen)	Final layer	Default start; small datasets (<300/class); distinct classes
Unfreeze last block	Last conv block + head	Validation plateaued; fine-grained classes; ≥500 images/class
Unfreeze last two blocks	More of the late backbone	Large-ish dataset; classes far from ImageNet's domain (medical, satellite)
Full fine-tuning	Everything, tiny LR	Rarely needed for student projects; risks overfitting on small data

Progressive unfreezing (head → last block → more) with validation checks at each stage is the disciplined approach, and it gives your report a clean methodology narrative: "freezing the backbone reached 0.87; unfreezing layer4 with LR 1e-5 reached 0.91; further unfreezing showed no gain."

The errors that waste the most time

Error / symptom	Cause	Fix
`RuntimeError: size mismatch` on the head	Replaced layer has wrong `in_features`, or class count wrong	Read `in_features` from the model (`model.fc.in_features`), don't hardcode
Accuracy stuck near 1/num_classes (random)	Forgot to unfreeze the head (everything frozen, nothing trains), or labels misaligned with `ImageFolder` class order	Check `requires_grad` on the head; print `full.classes` and verify folder-name → index mapping
Fine-tuning worse than frozen	Learning rate too high on pretrained layers	Drop to 1e-5 for backbone params; use per-layer LRs as shown above
Great training accuracy, poor validation	Overfitting — expected on small data	Augmentation, dropout on the head (`nn.Dropout(0.3)` before the final Linear), early stopping; see the overfitting guide
`model.eval()` forgotten at test time	Dropout/batchnorm still in training mode → noisy, wrong predictions	Always `model.eval()` before validation/inference; `model.train()` to resume training
Validation augmented	Augmented val transform applied to the validation set	Separate transforms; validation is resize + normalize only
Wrong normalization	Used 0–1 tensors or dataset-specific stats on a pretrained backbone	ImageNet mean/std, exactly as above

The model.eval() bug deserves special mention: it produces predictions that are slightly wrong and non-reproducible, which looks exactly like a "model quality" problem but is a one-line bug. When validation numbers look oddly noisy, check the train/eval mode first.

Evaluation: report it like the metrics guide demands

Transfer learning does not change what you report — the evaluation metrics guide applies in full: per-class precision/recall/F1, macro-F1, confusion matrix, and the test set touched once at the end. Two additions specific to transfer learning:

Report the ablation: frozen-backbone score vs fine-tuned score vs (if you tried it) from-scratch score. This table is the evidence that transfer learning earned its place, and it is the first thing a knowledgeable examiner asks for.
Show what the model looks at: Grad-CAM heatmaps overlaid on a few test images demonstrate that the model attends to the leaf lesion rather than the background. They are easy to generate for ResNet/MobileNet, they make excellent report figures, and they catch the embarrassing case where the model learned the background.

The TensorFlow/Keras equivalent in 20 lines

If your course or guide standardises on Keras rather than PyTorch, the same transfer-learning pattern applies — the API differs, the discipline does not:

from tensorflow.keras.applications import ResNet50
from tensorflow.keras import layers, models

base = ResNet50(weights="imagenet", include_top=False,
                input_shape=(224, 224, 3))
base.trainable = False  # freeze the backbone

model = models.Sequential([
    base,
    layers.GlobalAveragePooling2D(),
    layers.Dropout(0.3),
    layers.Dense(4, activation="softmax"),  # your class count
])
model.compile(optimizer="adam",
              loss="categorical_crossentropy", metrics=["accuracy"])

Fine-tuning in Keras: set base.trainable = True, then re-freeze everything except the last block by iterating base.layers and setting layer.trainable selectively, recompile with a small learning rate (Adam(1e-5)), and continue training. The critical Keras gotcha: you must call model.compile() again after changing trainable flags, or the change silently does nothing. BatchNormalization layers need special care during fine-tuning — keep them frozen (layer.trainable = False for BN layers) unless you understand why, because updating BN statistics on small batches destabilises training.

When transfer learning fails: the domain-gap problem

Transfer learning assumes the source domain (ImageNet photos) shares low-level structure with your domain. Usually true for natural images — plant leaves, dogs, birds, waste items are all photographs of the physical world. It breaks down when your images look nothing like photographs:

Medical scans (X-ray, MRI, histopathology): different textures, different colour spaces (often grayscale), different salient structures. Transfer learning still often helps as an initializer, but expect a smaller boost and plan to unfreeze more layers.
Satellite/aerial imagery: top-down geometry the backbone never saw. Works moderately; more fine-tuning needed.
Line drawings, sketches, diagrams: weak transfer. Consider training a smaller network from scratch with heavy augmentation instead.
Tiny images (32×32 thumbnails upscaled to 224): the backbone's features assume real resolution; upscaled thumbnails carry little signal. Fix the data pipeline before blaming the model.

The diagnostic is the same ablation table recommended earlier: if frozen-backbone scores barely beat a small from-scratch CNN, the domain gap is large and you should unfreeze aggressively or reconsider the approach. Do not keep a pretrained backbone out of habit when the numbers say it is not helping — the ablation exists to catch exactly this.

Folder structure and labelling discipline

ImageFolder (PyTorch) and image_dataset_from_directory (Keras) both assume the same layout:

data/
  train/
    healthy/      <- class folders; folder name = class name
    rust/
    blight/
    mildew/
  val/
    healthy/
    ...

Discipline rules that prevent the classic failures:

Folder names are your class labels. A typo ("healty") creates a phantom class. List dataset.classes and verify before training.
Split before augmenting, at the folder level. Copy (don't move) images into train/val/test folders with a script, seeded, so the split is reproducible and documented.
Keep the raw originals untouched in a separate raw/ directory. Every derived dataset (resized, cleaned, re-split) is generated by a script from raw. When an examiner asks "how did you split the data?", the answer is a script and a seed, not a memory.
A dataset-card.md in the project repo: source of images, count per class, labelling rules, known issues ("class 'blight' includes early and late stages; 12 images have watermark text"). This 15-minute document is the most impressive artifact in many vivas — it proves the data was engineered, not downloaded and hoped over.

PyTorch vs TensorFlow for this project: a straight answer

Both frameworks do transfer learning well; the choice should follow your constraints, not tribal loyalty:

Factor	PyTorch (+ torchvision)	TensorFlow/Keras
Learning resources for transfer learning	Enormous; most recent tutorials and papers	Enormous; most course-lab material
Debugging transparency	Eager execution; you can print any tensor mid-training	Eager too, but the compile step hides some errors until runtime
Deployment to phones/edge	Via ONNX/TorchScript export	TFLite is the smoothest edge path in the industry
Your guide's preference	—	If your lab or guide standardises on one, follow it — consistency beats preference

The tiebreaker most students ignore: use the framework your guide and labmates know. Debugging help at midnight comes from people, not documentation. If your entire lab is on Keras, the slightly nicer PyTorch API is not worth losing your support network.

Estimating training time before you commit

For your synopsis timeline, estimate honestly. A ResNet50 fine-tuning run on ~2,000 images (224×224) takes roughly 3–8 minutes per epoch on a Colab T4, depending on batch size and augmentation. A typical run — 15 epochs head-only plus 20 epochs fine-tuning — is therefore 2–5 hours of GPU time. Multiply by the number of experiments you plan (backbone comparison × 2, freezing strategies × 3, LR attempts × 2 = a dozen runs) and the project needs roughly 30–60 GPU hours total, spread across weeks. Colab free tier's daily limits handle this if you schedule one or two runs per day; it does not handle "train everything the night before the demo."

Two time-savers: cache the backbone features. If you are only training the head across many experiments, run the frozen backbone once over your dataset, save the feature vectors to disk, and train tiny heads on the cached features in seconds on CPU. And downscale while iterating — run architecture experiments at 160×160 or with a data subset, then confirm the winner at full resolution. Both are standard practice, and both belong in your methodology section as evidence of efficient experimentation.

Deployment notes

Export the trained model for the demo path: TorchScript or ONNX for laptop/server demos (the deployment guide covers serving it behind an API), or convert a MobileNetV2 to TFLite for on-device inference. Measure the actual inference time on the demo hardware and put the number in your report — "MobileNetV2 at 38 ms/frame on the Pi 4" is a result; "real-time" is a wish. For the full picture on keeping the whole pipeline honest — data, model, and demo — the testing and debugging guide is the companion to read next, and the ML project ideas guide if you are still scoping the project itself.

The report's methodology section: what it must contain

Examiners reconstruct your work from this section alone. For a transfer-learning project, it needs: dataset source and per-class counts; the split method (script, seed, ratios); backbone name and weight version; input size and normalization constants; augmentation list (train only); freezing strategy per stage with learning rates and epoch counts; optimizer and batch size; hardware and measured training time; and the evaluation protocol from the metrics guide (test set used once, per-class metrics, ablation table). If any item is missing, the work is not reproducible — and "reproducible from the text" is the bar. Write the section as you go, not at the end; reconstructing learning rates from memory a month later is how errors enter reports.

Build checklist

Dataset: target images per class with real variation; labels spot-checked; no train/val leakage.
Backbone chosen for the demo hardware (ResNet50 laptop / MobileNet edge), weights version noted.
ImageNet normalization and correct input size in the data pipeline; augmentation on train only.
Head replaced with the right class count; backbone frozen; only-head training converges (smoke test: loss falls in 3 epochs).
Fine-tuning attempted progressively with small LRs; each stage's validation score logged.
model.eval() before every validation/inference run.
Final evaluation: per-class metrics + confusion matrix on a held-out test set; ablation table (frozen vs fine-tuned).
Grad-CAM spot-checks confirming the model looks at the right regions.
Report documents the dataset, backbone, freezing strategy, LRs, and hardware — reproducible from the text.

Transfer learning turns an impossible data problem into a tractable one, but it is not magic: the data still needs to be clean, the evaluation still needs to be honest, and the freezing strategy still needs to be chosen by experiment rather than hope. Do those, and the pretrained backbone does the heavy lifting it was built for.

CNN Transfer Learning for Students: Fine-Tuning ResNet and MobileNet in PyTorch