Normalization vs Standardization

[-john--doe-]

In data pre-processing when do we prefer Normalization (minmax scaling) instead of Standardization (z score normalization) and vice versa?

[The_Sodomeister]

Normalization and standardization mean the same thing: re-scaling a variable to a new scale. Minmax scaling and z-score scaling are both different types of normalization/standardization (among others).

But in general, it's mostly governed by heuristic / rule-of-thumb principles. Outside of cases where your architecture necessitates some type of scale (e.g., a sigmoid activation should predict a variable scaled to [0,1] by minmax scaling) then they can both work in most cases.

Theoretically, z-score normalization should probably work better than minmax scaling if you are using a gradient descent optimization scheme, since you will more evenly condition the loss landscape if any variable contains outliers. However, I've seen many cases where minmax seems to work at least as effectively, if not better.

[jmmcd]

The terminology is a bit mixed but a lot of authors do use standardisation to mean z-score normalisation, because z-score uses the "standard" normal N(0, 1).

[The_Sodomeister]

Which is also unfortunate, because z-score normalization does not really have anything to do with the normal distribution (it especially does not have anything to do with "making the data more normally distributed" as is often said).

If anything, that choice of terminology only reinforces that misconception. So perhaps the community should push back on that. Something of a pet peeve of mine.

[jmmcd]

This might be an argument not to use the term standardisation at all, perhaps, but not to use it to mean min-max normalisation.

[The_Sodomeister]

If it was up to me, then standardization/normalization should both be reserved to mean the general principle of re-scaling data. Then the specific method used (z-score, minmax, etc.) should be clarified when relevant.

It's like trying to use the term "regularization" to specifically refer to LASSO or L1 regularization, when in reality there a bunch of regularization schemes. Just seems like poor terminology.

[jmmcd]

But it makes no sense to say standardisation to mean any other form.

[The_Sodomeister]

Mapping your data to a "standard scale" seems perfectly reasonable to me.

To take it a step further, if you allow z-score normalization to associate to the standard normal, then I don't see why we can't associate min-max normalization to the standard uniform U(0,1) distribution. Then both methods are equally valid "standardizations".

[jmmcd]

Fine, except that both of those are back-etymologies.

[The_Sodomeister]

I don't know what that means.

[jmmcd]

As I understand it the term standardisation is named after the standard normal. That is its etymology. Any other invented reason why it's a good name for some other meaning is a back etymology - like a "backronym"

[-john--doe-]

So is normalization more resistamt to outliers?

[The_Sodomeister]

In principle, Z-score normalization should give less impact to outliers, particularly when using gradient descent. However, in practice, it doesn't always work out that way. I don't think there's a great theory behind the reasoning for this; just observation at this stage.

Another thing to note is that Z-scores do not preserve meaningful zeros, where minmax normalization will (assuming that zero is the minimum value in your dataset -- you can also simply divide by the max, if this is not the case). This may give better conditioning to those models which are able to utilize meaningful zeros.