Feature Scaling Explained: Normalization vs Standardization

Feature Scaling Explained: Normalization vs Standardization and When to Use Each

A machine learning model sees numbers, not units. If one feature is age in years (roughly 18 to 60) and another is annual income in rupees (roughly 2,00,000 to 20,00,000), the income column is thousands of times larger. Many models will treat that size difference as importance, even though age might matter far more for the prediction. Feature scaling fixes this by putting every feature on a comparable scale before training.

Scaling is a small preprocessing step, but getting it wrong causes real, confusing failures: a k-nearest neighbours model that ignores the most useful feature, a neural network that trains slowly or not at all, or a model that works in the notebook and breaks on new data because the scaler was fitted incorrectly.

Quick answer

Use min-max normalization when you need every feature inside a fixed range, usually 0 to 1, for example before training a neural network or when a feature must stay bounded. Use standardization (z-score scaling) as the default for most other models: it centres each feature at mean 0 with standard deviation 1 and copes better with values outside the training range. Use robust scaling when the data contains genuine outliers that must not be allowed to dominate, because it scales with the median and interquartile range instead of the mean. Skip scaling entirely for tree-based models such as decision trees and random forests, which split on thresholds and do not compare feature magnitudes.

What feature scaling actually does

Scaling changes the numbers, not the information. It applies the same arithmetic rule to every value in a feature: subtract a centre, divide by a spread, or squeeze values into a range. The order of the values stays the same, and the relative gaps between them stay proportional (for the linear methods in this guide). What changes is the numeric range the model sees.

Three things scaling does not do. It does not change the shape of a distribution: a skewed feature stays skewed, just on a smaller scale. It does not remove outliers: an extreme value stays extreme relative to the rest. And it does not add information: if a feature was useless before scaling, it stays useless after.

Why unscaled features distort some models

Models that measure distance or use gradient steps are sensitive to scale. Models that split on thresholds are not.

Distance-based models include k-nearest neighbours, k-means clustering, and support vector machines. They compute distances between data points. In a distance sum, a feature ranging over thousands will swamp a feature ranging over tens, so the small-range feature barely influences which points count as neighbours.

A short illustration makes the point. Suppose a student project predicts house prices from two features: number of rooms (1 to 6) and floor area in square feet (400 to 3,000). Two houses differ by 2 rooms and by 500 square feet. In raw Euclidean distance, the 500 dominates the 2 completely, so the model effectively ignores room count. After scaling both features to a similar range, the two differences contribute comparably and the model can use both signals.

Gradient-based models include linear regression trained by gradient descent and neural networks. When features live on very different scales, the loss surface becomes stretched: steep in one direction, flat in another. Gradient descent then needs many small steps or oscillates, so training is slow and can fail to converge. Scaling makes the surface more even and lets the optimiser move sensibly in all directions at once.

Tree-based models (decision trees, random forests, gradient boosting) are largely unaffected. A split such as "age > 35" works the same whether age is measured in years or in scaled units, because the tree only cares about the ordering of values. Scaling a tree model's inputs is usually harmless but unnecessary work.

Min-max normalization: squeeze everything into 0 to 1

Min-max scaling maps the smallest value in a feature to 0, the largest to 1, and everything else proportionally in between. The rule is: scaled value = (value - minimum) / (maximum - minimum).

Use it when a fixed, bounded range matters. Neural networks train more stably when inputs sit in a small range, image pixel values are conventionally scaled from 0-255 to 0-1, and some algorithms and visualisations assume inputs inside 0 to 1. It is also the easiest method to explain and to invert: multiply back by the range and add the minimum to recover the original value.

Its weakness is outliers. A single extreme value stretches the range, so every ordinary value gets squeezed into a narrow band near one end. If one income in the data is ten times the next largest, min-max scaling will push almost all incomes below 0.1 and the model will struggle to tell them apart. Min-max also has no answer for a new value outside the training range: it simply lands outside 0 to 1, which can surprise a model that was trained to expect bounded inputs.

Standardization: centre at 0 with unit spread

Standardization, also called z-score scaling, subtracts the mean and divides by the standard deviation: scaled value = (value - mean) / standard deviation. The result has mean 0 and standard deviation 1. A value of +2 means two standard deviations above the mean; -1 means one below.

Use it as the default for most models that need scaling: logistic regression, SVMs, k-NN, k-means, PCA, and neural networks all behave well with standardized inputs. Because it uses the mean and standard deviation rather than the observed minimum and maximum, a new value outside the training range simply gets a larger z-score instead of breaking a boundary, which is usually the behaviour you want.

Its weakness is the same statistics it relies on: the mean and standard deviation are themselves pulled by outliers, so a heavy-tailed feature can still leave most values bunched together. Standardization also produces negative numbers and an unbounded range, which is fine for most models but unsuitable when an algorithm genuinely needs non-negative inputs.

Robust scaling: when outliers are real and must stay

Robust scaling replaces the mean with the median and the standard deviation with the interquartile range (the gap between the 25th and 75th percentiles): scaled value = (value - median) / interquartile range. The median and the interquartile range barely move when a few extreme values appear, so ordinary values keep a useful spread even in messy data.

Use it when outliers are genuine measurements rather than errors: sensor readings with occasional spikes, income or transaction data, or any feature where clipping the extremes would delete real signal. It is the safest choice when you have not yet inspected the data closely.

Its trade-off is that it is less familiar to readers of your report, and the scaled values have no standard range, so if a downstream step needs 0 to 1 inputs you will need another transform on top.

Comparison table

Method	Formula	Output range	Sensitive to outliers	Best for
Min-max normalization	(x - min) / (max - min)	0 to 1	Yes, strongly	Neural networks, image pixels, bounded inputs
Standardization (z-score)	(x - mean) / std	Unbounded, mean 0, std 1	Moderately	Default for SVM, k-NN, PCA, logistic regression
Robust scaling	(x - median) / IQR	Unbounded, median 0	No, by design	Data with genuine outliers, sensor readings
No scaling	—	Original units	—	Decision trees, random forests, gradient boosting

A four-question routine for picking a method

Is the model tree-based? If yes, stop: scaling is optional and usually skipped. Spend the effort on features instead.
Does the data contain outliers that are real? If yes, start with robust scaling and check the result before trying anything else.
Does the model or pipeline need inputs inside a fixed range? If yes, use min-max normalization, and check that outliers are handled first.
Otherwise, standardize. It is the most widely expected default, it handles new values gracefully, and it is what most tutorials and reviewers will assume you did.

Whatever you choose, apply the same method to every numeric feature that the model treats as comparable, and keep categorical features encoded separately: do not scale one-hot encoded columns, which are already 0 or 1.

Worked example in Python

The example below uses scikit-learn on a small student dataset with two features on very different scales: study hours per week (0 to 40) and mock-test score out of 100. It standardizes the features, fits a k-NN classifier, and shows the one rule that matters most: fit the scaler on the training data only.

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier

data = pd.DataFrame({
    "hours": [4, 12, 20, 8, 30, 15, 25, 6],
    "mock_score": [42, 58, 71, 49, 88, 63, 79, 45],
    "passed": [0, 0, 1, 0, 1, 1, 1, 0],
})

X = data[["hours", "mock_score"]]
y = data["passed"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=7
)

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

model = KNeighborsClassifier(n_neighbors=3)
model.fit(X_train_scaled, y_train)
print("Test accuracy:", model.score(X_test_scaled, y_test))

Note the two different calls: fit_transform on the training set, transform only on the test set. The test data never influences the mean and standard deviation. To swap in min-max scaling, replace StandardScaler with MinMaxScaler; for robust scaling, use RobustScaler. In a real project, wrap the scaler and the model in a scikit-learn Pipeline so the same steps run identically during training and prediction.

Common mistakes

Fitting the scaler on the full dataset before splitting. The test set then leaks into the training statistics, and the reported accuracy is optimistic. Always split first, fit on train only.
Fitting a separate scaler on the test set. Train and test then live on different scales, and the model receives inputs it was never trained on.
Scaling one-hot encoded columns. They are already 0 or 1; scaling them distorts the encoding and adds nothing.
Scaling the target variable by habit. Regression targets sometimes need a transform, but it is a separate decision with its own inverse step at prediction time. Do not scale the target just because you scaled the features.
Expecting scaling to fix skew. A right-skewed feature stays right-skewed after linear scaling. For skew, consider a log transform or a power transform first, then scale.
Scaling inside cross-validation the wrong way. The scaler must be fitted inside each fold, not once on the whole dataset. A Pipeline handles this automatically; manual scaling before cross_val_score leaks.

Frequently asked questions

Do I need to scale features for a random forest?

No. Trees split on thresholds, so the numeric range of a feature does not change where splits fall. Scaling is harmless but adds a step with no benefit.

Should I scale before or after the train-test split?

Split first, then fit the scaler on the training set only and transform both sets. Fitting before the split leaks information from the test set into training.

Which method should I use if I am unsure?

Standardization is the safest default for most models. If the data has clear outliers, try robust scaling and compare validation scores; if the model needs bounded inputs, use min-max.

Can scaling change my model's accuracy?

For distance-based and gradient-based models, yes, often substantially. For tree-based models, effectively no. If scaling changes a tree model's result, something else in the pipeline changed too.

Do I need to scale new data at prediction time?

Yes, with the same fitted scaler object saved from training. Never refit a scaler on the new data, and never feed raw values to a model trained on scaled ones.

Is normalization the same as standardization?

In casual use the words get mixed up, but in this guide they are distinct: normalization here means min-max scaling to a fixed range, standardization means the z-score transform. State which one you used in your report, because a reader cannot guess from the word "scaled" alone.

Limitations

Scaling is a preprocessing choice, not a modelling fix: it cannot rescue uninformative features, fix class imbalance, or correct a skewed distribution on its own. The guidance here covers the three linear methods students meet most often; non-linear transforms, quantile transforms, and per-feature pipelines are beyond this guide. Always report which scaler you used, which features it was applied to, and whether it was fitted on the training set only, because those three facts decide whether someone else can reproduce your results.

Feature Scaling Explained: Normalization vs Standardization and When to Use Each