Choose a Loss Function: MSE, Cross-Entropy & More

Every machine learning model learns by minimising a number, and that number is the loss function. It is the score the model tries to reduce during training: it measures how wrong the predictions are, and the optimiser follows its gradient downhill. Pick the wrong loss and the model trains happily toward the wrong goal — it can report a falling loss while producing useless predictions, or it can refuse to learn at all because the loss gives it no useful signal.

Choosing a loss is not a tuning detail you leave to a library default. The right choice depends on the task (regression or classification), the shape of the output, and the cost of different kinds of error. This guide covers the losses students actually need — mean squared error, mean absolute error and Huber for regression, cross-entropy in its three forms for classification — how to match each one to the output layer, a worked example you can run, and the mistakes that waste the most training time.

Quick answer

Use mean squared error (MSE) for ordinary regression, mean absolute error (MAE) when outliers should not dominate, and Huber when you want a compromise between the two. For classification use binary cross-entropy with a sigmoid output for two classes, categorical cross-entropy with a softmax output for one-of-many classes, and sparse categorical cross-entropy for the same multi-class case when your labels are integers instead of one-hot vectors. Always pair the loss with the matching output activation — a mismatch is the most common reason a student model fails to learn.

What a loss function does

Training is an optimisation problem. The model makes a prediction, the loss function compares it with the true answer and returns a single number, and the optimiser adjusts the weights a small step in the direction that reduces that number. Repeat millions of times and the model improves — but only in the sense the loss defines. The loss is the definition of "wrong", so it decides what the model cares about.

That is why two models trained on the same data with different losses behave differently. MSE punishes large errors much more than small ones, so a model trained with MSE will work hard to avoid a few big misses even at the cost of many small ones. MAE treats all errors equally, so the same model trained with MAE settles for a middle ground and ignores a few wild outliers. Neither is "correct" in the abstract — the right loss is the one whose definition of wrong matches what your project actually needs.

Three properties decide whether a loss works for your problem. First, it must be differentiable (or nearly so), because gradient-based optimisers need a slope to follow; a loss with no usable gradient gives the optimiser nothing to work with. Second, it must match the output: a loss built for probabilities makes no sense applied to raw scores, and vice versa. Third, it must penalise the errors you care about: if missing a rare fault matters more than a false alarm, the loss should say so, either directly or through class weights.

Regression losses: MSE, MAE and Huber

Regression predicts a number — a temperature, a price, a count — so the loss measures the gap between the predicted number and the true number.

Mean squared error (MSE) is the default, and the one most tutorials use without explanation. It squares each error, averages them, and the squaring does two things: it makes all errors positive, and it makes large errors count far more than small ones. An error of 10 counts a hundred times more than an error of 1, not ten times more. That is exactly right when a big miss is genuinely costly — predicting a temperature 10 degrees out is much worse than being 1 degree out — but it also means a few bad labels or sensor glitches in your training data can drag the whole model off course, because the optimiser spends its effort fixing those outliers. MSE is also the loss whose gradient grows with the error, which gives fast learning when the model is far from the answer and gentle fine-tuning when it is close.

Mean absolute error (MAE) averages the absolute size of the errors, so an error of 10 counts ten times more than an error of 1. It is robust: a handful of outliers cannot dominate training, because no single error is squared into importance. The trade-off is a less informative gradient — the slope of MAE is constant no matter how big the error is, so the optimiser gets no signal that a large error is more urgent than a small one, and training can crawl near the minimum. Use MAE when your data has noisy labels or occasional wild readings and you want the model to follow the typical case, not the worst case.

Huber loss is the compromise. It behaves like MSE for small errors and like MAE for large ones, switching at a threshold called delta (usually 1.0). Small mistakes get the strong, informative gradient of MSE; large mistakes get capped, so outliers cannot hijack training. Huber is a good default when you suspect outliers but cannot prove it, and it is the loss to try first when MSE training looks unstable because of a few bad samples.

Loss	Formula idea	Punishes large errors	Robust to outliers	Best when
MSE	Mean of squared errors	Strongly (squared)	No	Errors are roughly normal, big misses matter
MAE	Mean of absolute errors	Linearly	Yes	Data has noisy labels or wild readings
Huber	MSE near zero, MAE far away	Capped beyond delta	Mostly	You suspect outliers but want MSE-like learning

A quick calculation makes the difference concrete. Suppose the true values are 10, 20 and 30, and the model predicts 10, 20 and 40 — two exact, one off by 10. MSE is (0 + 0 + 100) / 3 = 33.3. MAE is (0 + 0 + 10) / 3 = 3.3. Now suppose instead the model is off by 5 on all three (predicts 15, 25, 35): MSE is (25 + 25 + 25) / 3 = 25, MAE is 5. MSE prefers the second model (25 vs 33.3) even though its average error is worse, because it hates the single big miss; MAE prefers the first model (3.3 vs 5). The loss you choose decides which of those two models your training produces.

Classification losses: cross-entropy in three forms

Classification predicts a category, so the model outputs probabilities and the loss measures how much probability it assigned to the wrong answers. Cross-entropy is the standard, and it has three forms that students constantly mix up.

Binary cross-entropy (BCE) is for two classes: yes/no, spam/not spam, fault/no fault. The model outputs a single probability through a sigmoid activation, and BCE penalises confident wrong answers heavily — predicting 0.99 for the wrong class costs far more than predicting 0.6. That steep penalty for confident errors is what makes the model calibrate its probabilities instead of guessing wildly. Use BCE whenever the answer is one of two classes.

Categorical cross-entropy (CE) is for one-of-many classes: digit 0–9, crop type, fault category. The model outputs one probability per class through a softmax activation, and CE looks only at the probability assigned to the true class — the other probabilities matter only through the softmax, which forces them to sum to one. Labels must be one-hot vectors (a 1 in the true class, 0 elsewhere). Use CE for any multi-class problem where each sample belongs to exactly one class.

Sparse categorical cross-entropy is the same loss as CE, but the labels are integers (0, 1, 2, ...) instead of one-hot vectors. The maths is identical; only the label format differs. It exists purely for convenience: with 100 classes, integer labels are far easier to store and pass around than 100-long one-hot vectors. If your labels are integers, use the sparse form; if they are already one-hot, use the ordinary form. Using the wrong one is the most common beginner error in this area — the training run crashes with a shape error, or worse, trains silently on misinterpreted labels.

A small Python example shows how the losses behave on the same predictions:

import numpy as np

def mse(y_true, y_pred):
    return np.mean((np.array(y_true) - np.array(y_pred)) ** 2)

def mae(y_true, y_pred):
    return np.mean(np.abs(np.array(y_true) - np.array(y_pred)))

def binary_cross_entropy(y_true, p):
    p = np.clip(p, 1e-7, 1 - 1e-7)  # keep log() away from 0
    return -np.mean(
        np.array(y_true) * np.log(p)
        + (1 - np.array(y_true)) * np.log(1 - p)
    )

print(mse([10, 20, 30], [10, 20, 40]))        # 33.33
print(mae([10, 20, 30], [10, 20, 40]))        # 3.33
print(binary_cross_entropy([1, 0, 1], [0.9, 0.1, 0.4]))  # 0.38
print(binary_cross_entropy([1, 0, 1], [0.9, 0.1, 0.9]))  # 0.08

The last two lines are the point: the same three labels score 0.38 when the model is unsure about the third sample (0.4) and 0.08 when it is confident and right (0.9). Cross-entropy rewards confidence only when the confidence is correct, which is exactly the behaviour you want from a classifier.

Match the loss to the output layer

The loss and the final activation are a pair. Get the pairing wrong and the model either fails to learn or learns a distorted version of the task.

Task	Output activation	Loss	Labels
Regression (one number)	Linear (none)	MSE, MAE or Huber	Real numbers
Binary classification	Sigmoid (one output)	Binary cross-entropy	0 or 1
Multi-class, one label	Softmax (one output per class)	Categorical cross-entropy	One-hot vectors
Multi-class, one label	Softmax	Sparse categorical cross-entropy	Integers
Multi-label (several labels at once)	Sigmoid (one output per label)	Binary cross-entropy per label	One-hot, multiple 1s

Two pairings deserve a warning. First, never use MSE for classification: it treats class labels as numbers, so it thinks class 2 is "between" class 1 and class 3, which is meaningless, and its gradient flattens when the model is confidently wrong, so learning stalls exactly when it is needed most. Second, never apply cross-entropy to raw logits (the numbers before softmax or sigmoid) unless the library function explicitly expects logits — most frameworks have a separate from_logits option for that case, and passing probabilities to a logits function (or vice versa) silently trains on nonsense. When in doubt, check whether your loss function expects probabilities or logits, and match the activation to that expectation.

Multi-label problems deserve a separate note, because students often force them into the multi-class shape. If a sample can carry several labels at once — an image containing both a cat and a dog, a project described by several tags — softmax is wrong, because it forces the probabilities to sum to one and makes the labels compete. Use one sigmoid output per label with binary cross-entropy on each, so each label gets an independent yes/no decision.

Special cases: imbalance, ranking and custom losses

Class imbalance. When one class is rare — a fault that appears in 2% of samples, a disease in 1% of patients — plain cross-entropy lets the model ignore the rare class entirely and still score a low loss by always predicting the common class. The fix is to make the rare class count more: pass class weights to the loss so errors on the rare class are multiplied, or use focal loss, which down-weights the easy, well-classified examples and forces the model to spend its capacity on the hard ones. Either way, judge the result with precision, recall and F1 on the rare class, not with overall accuracy — accuracy lies about imbalanced problems.

Ranking and similarity. If the task is to order items (search results, recommendations) or to measure similarity (face verification, duplicate detection), a per-sample loss is the wrong tool. Ranking uses pairwise or listwise losses that compare items against each other, and similarity uses contrastive or triplet loss, which pulls matching pairs together and pushes different pairs apart. These are beyond a first project, but recognise the shape: if the goal involves comparing samples rather than labelling them, the loss should compare samples too.

Custom losses. A custom loss is justified when the project's real cost is not captured by any standard loss — for example, when a false negative (missing a fault) costs ten times a false positive (a false alarm), you can write a loss that multiplies false-negative errors by ten. Keep custom losses simple, differentiable, and tested on toy numbers before training: compute the loss by hand for a few predictions and confirm it moves in the direction you expect. A custom loss that is wrong is worse than a standard loss that is merely approximate, because it trains the model confidently toward the wrong goal.

Common mistakes

Using accuracy as the loss. Accuracy is a metric, not a loss: it is not differentiable, so gradient descent cannot optimise it. Train with cross-entropy and report accuracy (and precision/recall/F1) as evaluation metrics.
MSE for classification. Covered above, but it is the single most common beginner mistake: the model trains, the loss falls, and the predictions are meaningless because the loss treats categories as numbers on a line.
Wrong label format for cross-entropy. One-hot labels with sparse categorical cross-entropy, or integer labels with ordinary categorical cross-entropy, both produce errors or silent mis-training. Match the loss to the label format.
Forgetting the activation pairing. Softmax with binary cross-entropy, or sigmoid with categorical cross-entropy, both break the probability interpretation the loss assumes. Check the pairing table before training.
Letting outliers choose the loss for you. If MSE training is unstable, check the data for bad labels before switching losses — a loss change can mask a data problem that will return in evaluation.
Judging a loss by its absolute value. A loss of 0.4 for one problem and 2.3 for another says nothing about which model is better; losses are only comparable within the same problem, loss and dataset. Judge models by task metrics on a held-out test set, not by the loss number.
Changing the loss and the model at the same time. If you change two things and the result improves, you do not know which one helped. Change the loss alone, retrain, compare on the same test split, and only then move on.

Frequently asked questions

Which loss should I use for my first project?
If it predicts a number, MSE. If it classifies into two classes, binary cross-entropy with sigmoid. If it classifies into several classes, categorical cross-entropy with softmax (or the sparse form for integer labels). These three cover nearly every first project, and you should only move beyond them when you can name the specific problem they fail to solve.

Can I change the loss halfway through training?
Technically yes, practically no. The loss defines the optimisation target, so changing it mid-training is like changing the destination halfway through a journey — the optimiser's progress so far was toward a different goal. Start a fresh training run with the new loss instead.

Why is my loss falling but my accuracy not improving?
The loss and the metric measure different things, and a falling loss with flat accuracy usually means the model is getting more confident about the answers it already gets right, or it is improving on easy examples while the hard ones stay wrong. Check per-class precision and recall, and look at the examples the model still gets wrong.

What is the difference between loss and cost?
In most student usage they are the same thing: the loss is the error for one sample (or one batch), and the cost is the average loss over the whole training set. Some courses use "cost function" for the averaged version and "loss function" for the per-sample version. The distinction rarely matters in practice.

Does the choice of loss affect training speed?
Yes. MSE gives large gradients for large errors, which speeds early learning but can overshoot near the minimum; MAE gives a constant gradient, which is steadier but slower to converge. Cross-entropy gives strong gradients when the model is confidently wrong, which is why classifiers usually train faster with it than with MSE. If training is slow, the loss is worth checking — but check the learning rate first, because that is the more common cause.

Should I use focal loss for every imbalanced dataset?
No. Start with class weights, which are simpler and easier to reason about. Move to focal loss only when class weights still leave the rare class under-detected, and remember that focal loss has its own tuning parameter (gamma) that needs its own experiments.

Limitations

This guide covers the losses a first or second ML project needs. It does not cover sequence losses (such as CTC for speech recognition), object-detection losses (which combine classification and bounding-box regression), or reinforcement-learning objectives, all of which deserve their own guides. The worked numbers in this guide are arithmetic illustrations on toy data, not experimental results — real loss values depend on the dataset, the model and the training setup, and the only honest way to compare two losses is to train with each on the same split and compare task metrics on a held-out test set. Finally, the loss is one piece of the training setup: a perfect loss cannot rescue bad data, a wrong model shape, or a broken training loop, so treat loss selection as one check in a longer list, not as a fix for everything.

How to Choose a Loss Function for Your ML Project: MSE, Cross-Entropy and More