In this guide
You changed the learning rate from 0.001 to 0.0005 and accuracy went up. Then you changed the batch size and it went down. Then you changed both and it went up again, and now you have no idea which change mattered, your Colab GPU hours are gone, and the model is somehow worse than the first version. This is what hyperparameter tuning looks like without a method — expensive guessing.
Hyperparameters are the settings you choose before training: learning rate, batch size, regularization strength, network size. Tuning them well is the difference between a model that learns and one that flails, but most students tune in the least efficient way possible: changing several things at once, judging on the test set, and stopping when they run out of patience. This guide gives you the practical method: what to tune in what order, which search strategies actually work, how to run a disciplined tuning loop on free compute, and when to stop tuning and fix your data instead.
What a hyperparameter is (and what isn't)
Parameters are learned by training — the weights and biases inside your model. Hyperparameters are chosen by you before training starts, and they control how learning happens. The ones that matter most in practice:
- Learning rate — the step size of gradient descent. The single most important hyperparameter in deep learning.
- Batch size — how many examples per gradient update. Interacts with learning rate and with your GPU memory.
- Regularization strength — weight decay, dropout rate, data augmentation intensity. These are your real levers against overfitting.
- Optimizer choice — Adam/AdamW vs SGD with momentum, and their settings.
- Architecture size — network depth and width, number of trees, and similar capacity knobs.
- Training duration — epochs, and when to stop.
Just as important: know what to leave at defaults initially. Adam's beta1/beta2, weight initializers, and similar second-order settings are well-tuned defaults — touching them before the big knobs is like adjusting the mirrors before learning to steer. Start from the framework or paper defaults for your model type, get a baseline training, then tune the list above in order.
Rule zero: build a trustworthy baseline first
No tuning method can save you without three things in place:
- A fixed train/validation/test split. Tune on validation, evaluate on test exactly once at the end. Every time you peek at the test set to pick a hyperparameter, it stops being a test set — your reported number becomes optimistic fiction. (More on this discipline below.)
- One metric you actually care about, tracked on both train and validation. The train-vs-validation gap is your diagnostic instrument: it tells you whether you're underfitting, overfitting, or just badly tuned.
- A log of every experiment. A simple CSV or notebook table: hyperparameters tried, train metric, validation metric, notes. Memory is not a lab notebook — after ten runs you will misremember which change helped.
Train one baseline with default settings and record it. That baseline is the number every future experiment must beat, and it tells you whether you have a tuning problem at all.
What to tune, in order of impact
1. Learning rate. Too high and training diverges or oscillates — the loss explodes or bounces. Too low and it crawls, wasting your Colab hours and settling into mediocre solutions. Learn to read the loss curve: a healthy run drops fast early, then settles into a noisy decline; divergence means lower the rate; a flat-then-slowly-falling curve often means you can raise it. When in doubt, try values spaced by factors of 3–10 (1e-2, 3e-3, 1e-3, 3e-4): learning rate effects are logarithmic, so 0.001 vs 0.0012 is never the experiment that matters.
2. Batch size. Larger batches give smoother gradient estimates and better GPU utilization; smaller batches add noise that can act as mild regularization. Practical rule: use the largest batch that fits your GPU memory, then tune the learning rate around it. If you must shrink the batch, expect to lower the learning rate somewhat too — the two interact.
3. Regularization. If validation performance stalls while training performance keeps improving, you don't have a learning-rate problem — you have an overfitting problem, and the fix is regularization: increase weight decay, add or raise dropout, strengthen data augmentation. These knobs are covered in depth in the overfitting guide linked below; tuning them is where most "my model won't generalize" cases are actually solved.
4. Optimizer. Adam or AdamW with defaults is the right starting point for almost all student deep-learning work — it's forgiving of learning-rate choice. SGD with momentum sometimes reaches a better final result but is fussier to tune. Don't switch optimizers until the learning rate and regularization are roughly right; changing the optimizer resets what you learned about the other knobs.
5. Architecture size. A bigger model can memorize more, which helps underfitting and hurts overfitting. But architecture changes are the most expensive experiments and usually the least efficient lever: most student models are limited by data quality and quantity long before they're limited by capacity. Tune the first four before reaching for a bigger network.
Search strategies, from simplest upward
Manual (coordinate) tuning: change one hyperparameter, keep the log, compare against baseline. Slow but deeply informative — you learn how each knob behaves on your problem. Every student should do at least a few rounds of this before automating.
Grid search: try every combination of a few values per hyperparameter. Sounds thorough, wastes compute: with 3 values each for 4 hyperparameters you run 81 trainings, and most of the grid explores the unimportant dimensions. Grid search is only sensible for 1–2 hyperparameters with a handful of values each.
Random search: sample each hyperparameter randomly from a range (log-uniform for learning rate and weight decay — sample the exponent, not the value). For the same budget as a grid, random search explores far more distinct values of the important dimensions, because every trial differs in every hyperparameter instead of repeating values. This is the best default strategy for students: simple, parallelizable, and surprisingly hard to beat.
Successive halving: built for tight compute budgets. Start many configurations but train each for only a few epochs; keep the top half, train longer; repeat. Bad configurations reveal themselves early — a learning rate that diverges at epoch 3 won't magically recover at epoch 50. This fits Colab's free tier beautifully: breadth first, depth only for winners.
Early stopping: not a search strategy but a free one — monitor validation loss and stop training when it stops improving (with a patience of several epochs). It converts "how many epochs?" from a hyperparameter you must tune into a decision the run makes itself, and it's mild regularization into the bargain.
Bayesian optimization: the conceptual idea — fit a cheap surrogate model that predicts how good a hyperparameter setting will be, based on trials so far, then pick the next trial balancing "likely good" against "unexplored" (exploration vs exploitation). Libraries like Optuna implement this in a few lines. Reach for it when each training run is expensive and you've exhausted manual and random search — not as your first tool.
A minimal random-search loop you can actually run — log-uniform learning-rate sampling with a CSV log:
import csv, math, random
def run_trial(lr, weight_decay):
# build model, train, return validation accuracy
...
return val_accuracy
with open("tuning_log.csv", "w", newline="") as f:
log = csv.writer(f)
log.writerow(["lr", "weight_decay", "val_accuracy"])
for trial in range(12):
lr = 10 ** random.uniform(-4, -2) # 1e-4 to 1e-2, log-uniform
wd = 10 ** random.uniform(-5, -3) # 1e-5 to 1e-3, log-uniform
acc = run_trial(lr, wd)
log.writerow([f"{lr:.2e}", f"{wd:.2e}", f"{acc:.4f}"])
print(f"trial {trial}: lr={lr:.2e} wd={wd:.2e} val_acc={acc:.4f}")
Twelve trials, one script, every result recorded. That alone puts you ahead of most student tuning.
A Colab-budget tuning workflow
Free GPUs mean your budget is measured in runs, not hours. Spend it like this:
- Freeze everything except 2–3 knobs. Learning rate and weight decay first; everything else at defaults.
- Run short proxy experiments. A quarter of your planned epochs is usually enough to rank configurations — relative ordering stabilizes long before final convergence. Confirm only the winner with a full run.
- Successive halving by hand: 8 configs x short runs, keep 4, train longer, keep 2, full run on the best.
- Fix the random seed for comparisons so differences come from hyperparameters, not luck — then re-run the winner with 2–3 different seeds to check the improvement is real, not seed noise.
- Checkpoint the best model of every run (save on best validation score, not last epoch) so a crashed session never costs you the result.
Validation discipline: the rule that makes tuning honest
This is the section that separates real results from self-deception:
- Tune on validation, report on test, once. The test set is locked away until every hyperparameter decision is final. Then you evaluate once and report that number — whatever it is.
- Repeated test-set peeking is leakage. Each peek lets test information steer your choices; after a dozen peeks your "test accuracy" is just another validation score wearing a costume, biased upward.
- Small datasets need cross-validation. With a few hundred examples, a single validation split is noisy — the winning hyperparameters might just be lucky on that split. K-fold cross-validation averages the noise away; the technique is explained in the cross-validation guide below.
- Keep a held-out sanity slice if you can: data collected or labeled after tuning started, never touched during it. If performance collapses on it, something leaked.
Worked example: tuning a small CNN (the process, not magic numbers)
Setup: an image-classification CNN trained from scratch on a small public dataset, baseline with framework defaults, train/val/test split fixed, accuracy logged per epoch.
Trial 1 (baseline, defaults): trains fine, validation accuracy plateaus well below training accuracy. Diagnosis: overfitting, not a learning-rate problem — tuning the learning rate now would be wasted effort.
Trials 2–4 (regularization sweep): increase weight decay across three log-spaced values, keep everything else fixed. Validation improves, train-val gap narrows. One of the three is clearly best; the strongest value starts hurting training accuracy — that's the signal you've gone too far.
Trials 5–8 (learning rate around the winner): try 3x higher and 3x lower than default with the chosen weight decay. The higher rate diverges early (killed by successive halving after a few epochs — cheap lesson); the lower rate trains stably but finishes slightly worse than default. Default rate confirmed.
Trials 9–10 (augmentation): add standard flips/crops. Validation improves again — augmentation was the bigger lever than any learning-rate tweak.
Final: re-train the winning configuration with three different seeds, evaluate on the test set once, report the mean. Total: about ten short runs and three full ones — an afternoon on a free GPU, and every decision is written in the log with its evidence.
Notice what didn't happen: no grid over six hyperparameters, no test-set peeking to pick winners, no claiming victory from a single lucky seed.
When tuning is a waste of time
Tuning optimizes the learning process; it cannot fix a broken learning setup. Stop tuning and look elsewhere when:
- The data is the problem. Mislabeled examples, train/test mismatch, or simply too little data — no hyperparameter teaches a model what isn't in the data. Data work (cleaning, augmentation, feature engineering) beats tuning by an order of magnitude here.
- There's no baseline. If you can't say what the current score is and how it's measured, you're not tuning, you're wandering.
- The train/val/test discipline is broken. Tuning against the test set manufactures fake improvements that evaporate on new data.
- You're past the flat part of the curve. The first few sensible trials capture most of the gain; trial 40 is rarely meaningfully better than trial 8. Diminishing returns are real — spend the remaining budget on data or on writing up.
- The model fundamentally underfits. Training accuracy itself is poor: the model is too small, the features too weak, or the task mis-specified. Capacity and representation first, tuning second.
FAQ
How many trials do I need?
Enough to cover the important dimensions coarsely: 8–15 random trials over 2–3 hyperparameters resolves most student tuning problems. If the best trial sits at the edge of your search range, extend the range and run a few more — the optimum may lie outside what you sampled.
Should I tune on Colab's free GPU or buy compute?
Free tier is plenty for student-scale tuning if you use short proxy runs and successive halving. Pay for compute only when a single full training run takes so long that even proxy experiments don't fit in a session — that's a sign to shrink the model or dataset first.
Grid search vs random search — which one for my project report?
Random search, and say why in the report: for a fixed budget it explores more distinct values of the influential hyperparameters. Examiners respond well to a log-uniform random search with a CSV log — it shows method, not luck.
Can I use Optuna or Keras Tuner instead of hand-rolling?
Yes — both are excellent, and Optuna's Bayesian sampler is a genuine upgrade over random search for expensive runs. But understand what the tool is doing first (the strategies above), or you'll misconfigure the search space and blame the tool for the results.
My validation accuracy jumps around between runs with the same settings. What do I do?
That's seed noise, especially on small datasets. Fix the seed when comparing hyperparameters, then verify the winner across 2–3 seeds. If the "improvement" disappears across seeds, it was never real — widen the data or the effect, not the tuning.
Is a higher learning rate with warmup better than a lower constant one?
Learning-rate schedules (warmup, decay, cosine annealing) often beat a constant rate, but they're a second-order concern. Get the base learning rate and regularization right first; add a schedule when the basics are solid and you want the last few percent.
Limitations
- No free lunch: no tuning strategy is best for every problem; the "best" settings are always relative to your dataset, model and metric.
- Tuning can't fix data problems — label noise, leakage, distribution shift and tiny datasets dominate hyperparameter effects.
- Compute is the real budget. Every strategy here trades runs for insight; on free tiers, discipline (proxy runs, halving, early stopping) matters more than the choice of optimizer.
- Reproducibility is partial. Same code plus same seed reproduces; different hardware or library versions may not, bit-for-bit. Report seeds and versions.
- Goodhart risk: optimizing a single metric hard enough can distort it — keep an eye on secondary metrics (and on what the model actually gets wrong) alongside the headline number.
Keep reading
- Cross-Validation Explained for ML — the honest way to tune on small datasets.
- Fix Overfitting in Machine Learning — regularization, the lever you'll tune most.
- ML Evaluation Metrics Explained — pick the right metric before you optimize it.
- Feature Engineering Basics — when better features beat more tuning.