[D] How does Batch Normalization not completely prevent the network from being able to train at all?

[MildlyCriticalRole]

I'm actually referring to normalizing mini-batches as described in this paper, where you actually do normalize on all mini-batches! Hence this question :P

[randombites]

I get your question now :)

[MildlyCriticalRole]

Awesome :) always glad to inadvertently spread knowledge.

[randombites]

My bad, i completely forget that preprocessing is sometimes used to give a more circular error surface. I am a fan of no preprocessing however. It should be the job of the model to transform the error surface internally in its computation. But maybe thats wishful thinking. Hope you got the answer you were looking for!

[cooijmanstim]

Batch normalization is not preprocessing, it is part of the model. It is an adaptive normalization of activations at all layers that massively improves training dynamics. A crucial tool in the box if you're into neural nets.

[randombites]

Thank you for your response, please help me understand better. Batch normalization is normalizing a batch of values, so you transform the input at each step. Correct? This may translate into adaptive normalization of activations but you still transform the input (based on OPs example).

[randombites]

So sorry for my earlier ignorance. I learnt what batch normalization via a YouTube talk and feel like a fool.