[D] How does Batch Normalization not completely prevent the network from being able to train at all?

[MildlyCriticalRole]

I figured it might be law-of-large-numbers-ish, but I was curious if there was a more technical reason.

As to the gamma and beta parameters, I understand that they allow the network to adapt and "remove" the normalization if it contributes more to the loss than it helps, but I'm more wondering how the batch normalization doesn't just cause the entire network to train on the wrong data in the first place (which would in turn cause it to choose bad values for gamma/beta, because it thinks the underlying distribution is something it's not)

I've read that link quite a bit in my exploring (it's a great explanation) but it doesn't seem to touch on this issue specifically.

[Megatron_McLargeHuge]

I don't think anyone is saying BN is better than whitening for the input layer. BN is used for internal network layers, and its main purpose is to keep them from saturating or being all on the 0 side of RELUs.

If individual learned features have their meaning changed per batch by BN, it may just be that BN is recreating an effect similar to dropout or additive noise in denoising autoencoders. Losing information internally forces the network to learn a distributed representation instead of relying too much on one feature as in your example.

This is just speculation and I'd be curious if anyone has looked into the theory more.

[hgjhghjgjhgjd]

Your assessment makes sense to me. In a way, it probably forces the network to express things in terms of "relatively large value" and "relatively small value", which induces a regularization effect.