In this guide
Gradient Descent Explained: Learning Rate, Momentum and Adam
Quick Answer
Gradient descent trains a model by nudging its parameters downhill on the loss surface: each step moves opposite the gradient, scaled by the learning rate. Momentum smooths the path by carrying velocity through flat spots and ravines, while Adam adapts a separate step size for every parameter. Start with Adam, tune the learning rate first, and trust the loss curve.
| Optimiser | Core idea | Best starting point |
|---|---|---|
| Plain SGD | One fixed step size for every parameter | Small models, full control |
| SGD + momentum | Carries velocity through flat spots and ravines | Deep networks, noisy gradients |
| RMSProp | Adapts the step size per parameter from recent squared gradients | Recurrent networks, non-stationary targets |
| Adam | Momentum plus RMSProp, with bias correction | Most models — the safe default |
| AdamW | Adam with weight decay handled separately | Any training that uses regularisation |
What gradient descent actually does
A model learns by minimising a loss function — a single number that says how wrong its predictions are. Gradient descent finds the parameters that make that number small: at each step it asks in which direction the loss increases fastest (that direction is the gradient) and moves the opposite way:
θ ← θ − η × ∇L(θ)
Here θ stands for the model's parameters, ∇L(θ) is the gradient of the loss, and η (eta) is the learning rate — the size of each step. The gradient gives the direction; the learning rate gives the distance. Get either one wrong and training stalls, oscillates, or diverges entirely.
Think of the loss surface as a landscape: a model with two parameters lives on a two-dimensional map with hills and valleys, and a real neural network lives in millions of dimensions, but the picture still holds. Gradient descent is a walker who can only feel the slope underfoot and always steps downhill.
Gradient descent finds a local minimum, not necessarily the global one. For convex problems such as linear regression there is only one valley, so any minimum is the best one. For neural networks the surface is full of valleys, saddle points and plateaus, and in practice a good local minimum is enough.
Choosing the loss function matters as much as the optimiser: the gradient is the gradient of something, and a poorly chosen loss gives misleading directions no matter how good the update rule is. If you have not settled yours yet, read how to choose a loss function for an ML project.
Batch, stochastic and mini-batch gradient descent
The gradient can be computed over different amounts of data, and that choice changes training behaviour.
Batch gradient descent computes the gradient over the entire training set before taking a single step. Each step is accurate and smooth, but on a large dataset one step can take minutes, and it can get stuck oscillating across a narrow valley.
Stochastic gradient descent (SGD) computes the gradient from a single example. Each step is cheap and fast, and the noise can help the walker escape shallow dips. The price is a jittery path: the loss bounces around instead of falling smoothly, and judging when to stop becomes harder.
Mini-batch gradient descent is the compromise almost everyone uses in practice. The gradient is averaged over a small batch — typically 32, 64 or 128 examples — which smooths out most of the noise while keeping each step cheap. It also lets the hardware work efficiently: a GPU processes 64 examples almost as fast as one. When people say "SGD" in modern deep learning, they almost always mean mini-batch SGD.
| Variant | Gradient from | Step quality | Speed per step |
|---|---|---|---|
| Batch | Whole dataset | Exact, smooth | Slow |
| Stochastic | One example | Noisy, jumpy | Fast |
| Mini-batch | 32–128 examples | Balanced | Fast, hardware-friendly |
The learning rate: why it is relative to curvature
The learning rate is the single most important hyperparameter in most training runs, and the most misunderstood. It is not an absolute speed — it is a step size relative to the curvature of the loss surface. On a gentle slope, a step of 0.1 barely moves anything; on a steep, sharply curved valley, the same 0.1 can throw the walker clean over the far wall.
The cleanest way to see this is a one-dimensional parabola, where every number can be checked by hand. Take L(θ) = (θ − 3)² + 1: its minimum is at θ = 3, where the loss is 1, and its curvature (second derivative) is 2 everywhere. The gradient is 2(θ − 3), so the update is θ ← θ − η × 2(θ − 3). Start at θ = 0:
| Learning rate η | θ after 5 steps | Loss after 5 steps | What happened |
|---|---|---|---|
| 0.01 | 0.2882 | 8.3537 | Crawling — barely moved from the start |
| 0.1 | 2.0170 | 1.9664 | Steady progress toward the minimum |
| 1.0 | 6.0, then 0, then 6... | 10.0 forever | Oscillating across the valley, never settling |
| 1.2 | 19.1347 and growing | 261.3292 | Diverging — each step overshoots further |
These numbers were computed directly from the update rule, not estimated. At η = 0.1 the first five steps are θ = 0.6, 1.08, 1.464, 1.7712, 2.017, with the loss falling 6.76, 4.6864, 3.3593, 2.5099, 1.9664. At η = 0.01 the walker reaches only θ = 0.2882 after five steps. At η = 1.0 it bounces between 0 and 6 forever, because the step exactly doubles the distance from the minimum each time.
The pattern is governed by a hard edge: for a quadratic loss with curvature c, plain gradient descent converges only while η < 2/c. At exactly η = 2/c it oscillates forever; above that it diverges. Here c = 2, so the edge sits at η = 1 — which is why 1.0 oscillates and 1.2 explodes. Real surfaces have different curvatures in different directions, so the safe rate is set by the steepest direction, which is why deep networks train at rates like 0.001 or 0.01.
Two symptoms diagnose the rate: a loss that falls very slowly or plateaus early means it is too small; a loss that jumps up, spikes or turns to NaN means it is too large. Between those extremes there is a wide workable range. A quick range test (start tiny, raise it steadily, plot the loss) finds it in minutes.
The learning rate also interacts with feature scale: if one input ranges from 0 to 1000 while another ranges from 0 to 1, the loss surface stretches into a narrow valley and the safe rate shrinks. Scaling first makes the valley rounder — see feature scaling: normalisation vs standardisation.
Momentum: velocity, not just a bigger step
Plain gradient descent forgets everything at each step. Momentum gives the walker velocity: instead of stepping straight from the current gradient, it keeps a running average of past gradients and steps in that smoothed direction.
v ← β × v − η × ∇L(θ), then θ ← θ + v
The momentum coefficient β is usually 0.9. Each new gradient contributes only 10% to the velocity while the accumulated history contributes 90%, so the walker builds up speed where the gradient keeps pointing the same way and cancels out directions where it flips.
That solves the classic failure of plain descent: the long narrow valley, or ravine. In a ravine the loss is steep across the valley and gentle along it, so plain descent zig-zags violently across the walls while creeping slowly along the floor. Momentum averages those opposing cross-valley gradients away, so the zig-zag shrinks, while the gentle along-valley gradients accumulate into real speed.
Two cautions. First, momentum is not a free speed-up: in a steady direction the effective step grows by roughly 1/(1 − β), which is 10× at β = 0.9, so the learning rate usually needs lowering when momentum is added. Second, velocity can overshoot — a walker arriving at the minimum with speed keeps going past it, as the NumPy example below shows. Momentum earns its keep in ravines and noisy landscapes, not on every problem.
RMSProp: a different step size for every parameter
Some parameters need big steps and others need tiny ones: a parameter tied to a rare feature may see a meaningful gradient once in a hundred batches, while one tied to a common feature sees one every batch. A single global learning rate cannot serve both.
RMSProp (root mean square propagation) fixes this by tracking a moving average of each parameter's squared gradient, then dividing the step by the square root of that average:
E[g²] ← ρ × E[g²] + (1 − ρ) × g², then θ ← θ − η × g / (√E[g²] + ε)
The decay ρ is usually 0.9, and ε (a tiny number like 1e-8) only stops division by zero. The effect: a parameter with consistently large gradients gets its steps shrunk, and one with small, rare gradients gets its steps boosted. Each parameter ends up moving at roughly the same pace, which is why RMSProp was popular for recurrent networks, where gradients vary wildly across time steps.
Adam: momentum plus RMSProp, with bias correction
Adam (adaptive moment estimation) is the default optimiser in most frameworks: it combines the two ideas above, keeping a moving average of the gradient itself (the first moment, like momentum) and of the squared gradient (the second moment, like RMSProp), then using both to set the step.
m ← β₁ × m + (1 − β₁) × g
v ← β₂ × v + (1 − β₂) × g²
m̂ = m / (1 − β₁^t), v̂ = v / (1 − β₂^t)
θ ← θ − η × m̂ / (√v̂ + ε)
The standard settings are β₁ = 0.9, β₂ = 0.999, ε = 1e-8, and a starting learning rate of 1e-3. The bias-correction lines (dividing by 1 − β^t, where t is the step number) matter more than they look: both moving averages start at zero, so early on they underestimate the true moments, and without correction the first steps would be far too small. After a few dozen steps the factor is close to 1 and fades into the background.
Adam's strength is that it needs little tuning to get a model training at all. Its weakness is subtler: its adaptive steps can settle into sharp minima that generalise slightly worse than the flatter minima SGD with momentum tends to find — an empirical observation from several large vision models, not a theorem. The practical rule: start with Adam, and if final accuracy matters, try SGD with momentum as a comparison, judged on held-out validation, not training loss. For a disciplined way to run that comparison, see hyperparameter tuning: a practical student guide.
AdamW: one paragraph, because the difference matters
Adam has a quiet flaw in how it handles weight decay (L2 regularisation). In standard Adam, the decay term is added to the gradient before the adaptive scaling, so a parameter with large gradients gets its regularisation shrunk and one with small gradients gets it amplified — the opposite of what regularisation should do. AdamW fixes this by decoupling: the decay is applied directly to the parameters, outside the adaptive update. Same optimiser, same defaults, correct regularisation. If you use weight decay at all, use AdamW instead of Adam.
Learning-rate schedules: warmup and cosine decay
A fixed learning rate is a compromise: large early for fast progress, small late for a careful finish. A schedule removes it by changing the rate over the course of training.
Warmup starts the rate near zero and raises it linearly over the first few percent of training. Early gradients are noisy — the model is random and the surface is at its most chaotic — so large early steps can throw training off before it begins. Warmup is cheap insurance for large models.
Cosine decay lowers the rate smoothly along a cosine curve from its peak to near zero by the end of training. The gentle tail lets the model settle precisely into a minimum instead of bouncing around it. It is the most common schedule for a fixed-length run.
A common recipe: a short warmup (about 5% of steps), then cosine decay to zero. If the run is interrupted and restarted, restart the schedule too. And never judge a schedule by training loss alone — the careful finish exists for validation performance, which cross-validation explained for ML projects helps you measure.
Worked example: a real NumPy training loop
The snippet below fits a straight line y = 2x + 1 to five toy points, first with plain SGD and then with momentum 0.9, both at a learning rate of 0.1 for 60 epochs. It was run as written, and the printed numbers are the actual output:
import numpy as np
x = np.array([0.0, 1.0, 2.0, 3.0, 4.0])
y = np.array([1.0, 3.0, 5.0, 7.0, 9.0]) # true line: y = 2x + 1
def loss(w, b):
return float(np.mean((w * x + b - y) ** 2))
def grads(w, b):
err = w * x + b - y
return float(2 * np.mean(err * x)), float(2 * np.mean(err))
def train(momentum, lr=0.1, epochs=60):
w, b, vw, vb = 0.0, 0.0, 0.0, 0.0
for epoch in range(1, epochs + 1):
gw, gb = grads(w, b)
vw = momentum * vw - lr * gw
vb = momentum * vb - lr * gb
w, b = w + vw, b + vb
if epoch in (10, 30):
print(f" epoch {epoch}: loss = {loss(w, b):.4f}")
return w, b, loss(w, b)
print("initial loss:", round(loss(0.0, 0.0), 4))
for name, mom in [("plain SGD", 0.0), ("SGD + momentum 0.9", 0.9)]:
print(name)
w, b, final = train(mom)
print(f" final: w = {w:.4f}, b = {b:.4f}, loss = {final:.6f}")
Output:
initial loss: 33.0
plain SGD
epoch 10: loss = 0.0069
epoch 30: loss = 0.0006
final: w = 2.0023, b = 0.9934, loss = 0.000015
SGD + momentum 0.9
epoch 10: loss = 9.0877
epoch 30: loss = 0.5231
final: w = 1.9886, b = 1.0024, loss = 0.000674
Both methods recover the true line (w ≈ 2, b ≈ 1) from a cold start. But look at the trajectories: plain SGD is already at a loss of 0.0069 by epoch 10, while momentum is still at 9.0877 — its accumulated velocity overshoots the minimum badly at first, then spends the next 50 epochs recovering. By epoch 60 momentum reaches 0.000674, still behind plain SGD's 0.000015.
That surprises people who expect momentum to always win. This toy problem is a gentle, well-conditioned bowl — exactly the landscape where plain descent is already efficient. Momentum's advantage appears in ravines and noisy gradients, where plain descent zig-zags or stalls. The lesson is not "momentum is worse"; it is that no optimiser wins everywhere.
Which optimiser should you use?
| Situation | Start with | Why |
|---|---|---|
| First model, just get it training | Adam, lr 1e-3 | Works with minimal tuning |
| Image classification, large dataset, time to tune | SGD + momentum 0.9, lr 0.01–0.1, cosine decay | Often generalises slightly better — an empirical observation, not a guarantee |
| Recurrent network or uneven gradient scales | Adam or RMSProp | Per-parameter adaptation handles the variance |
| Training with weight decay | AdamW, not Adam | Decoupled decay regularises correctly |
| Tiny model, convex problem (linear/logistic regression) | Plain SGD or batch descent | No need for adaptivity; the surface has one valley |
| Loss exploding or NaN | Lower the learning rate first, then check the data | The optimiser is rarely the real cause |
5 common mistakes
- Tuning everything except the learning rate. Architecture, batch size, dropout and optimiser choice all get attention while the learning rate stays at its default. It is the most influential setting in most runs — test it first.
- Comparing optimisers on training loss only. Adam frequently reaches a lower training loss and still scores worse on held-out data. Every comparison that matters is on validation performance.
- Forgetting to shuffle. Training on data in its original order — all of one class, then all of another — feeds the optimiser a biased gradient at every step. Shuffle every epoch; in PyTorch and TensorFlow this is one flag on the data loader.
- Using weight decay with Adam instead of AdamW. Adam folds the decay into the adaptive scaling and distorts it. The fix is a one-word change to AdamW, and forgetting it silently weakens your regularisation.
- Changing two things at once. A new optimiser and a new learning rate in the same run tells you nothing about which one helped. Change one variable, record the validation score, then change the next.
Frequently asked questions
What learning rate should I start with?
For Adam, 1e-3 is the standard starting point; for SGD with momentum, 0.01 is a reasonable first try on a small network. Then run a range test: if the loss falls steadily you are in the right region; if it spikes or goes to NaN, divide by ten and retry.
Is Adam always better than SGD?
No. Adam is easier to get training and is the right default, but SGD with momentum, properly tuned, often generalises slightly better on large vision models — an empirical observation, not a rule. For any specific model, try both and compare on validation data.
Why does my loss go up instead of down?
Almost always the learning rate is too high: each step overshoots the minimum and lands higher than it started. Halve the rate and watch. If it still rises, check the data pipeline.
What is the difference between momentum and Adam?
Momentum adds velocity from past gradients to smooth the path; Adam adds that plus a per-parameter step size based on recent squared gradients, with bias correction for the early steps. Adam is momentum and RMSProp combined, not a different family.
Should I use a learning-rate schedule?
For any run longer than a quick experiment, yes — a short warmup followed by cosine decay to zero is a solid default. For a five-minute test run, a fixed rate is fine.
How do I know if my learning rate is too high?
The loss jumps upward, oscillates wildly, spikes, or becomes NaN within the first few epochs. A rate that is too low looks different: the loss creeps down slowly and plateaus far above where it should.
Limitations
Gradient descent is a local search method, so it can settle in a poor local minimum or crawl across a saddle point — no optimiser removes that. Adam's adaptive steps add memory and per-parameter state, and can behave unpredictably on very small datasets or badly scaled inputs. The worked examples here use a toy problem with five points, so treat every rule here as a starting point to verify on your own loss curves.
Need help with your model?
If you are stuck choosing an optimiser, tuning a learning rate, or getting a model to train at all, message Projectech on WhatsApp at wa.me/918275719193 with your base-paper or model details and your deadline. The reply is a quotation stating the scope, the price and the delivery timeline.