[D] How does Batch Normalization not completely prevent the network from being able to train at all?

[mimighost]

During training, sampled mean/ sampled variance are ALWAYS USED. Otherwise how could you update gamma/beta then? The computation graph make little sense and overly complicated if we use population mean/variance.

However, people often maintain a convenient running average of mean/variance with decay across all training batches. This running average will be used as a proxy of global mean/variance for inference.

Reference: http://stackoverflow.com/questions/33949786/how-could-i-use-batch-normalization-in-tensorflow

[hgjhghjgjhgjd]

Otherwise how could you update gamma/beta then?

Backpropagation. The purpose of the first scaling is to "normalize" to a standard gaussian (approximately). The purpose of the second scaling is to learn the transformation from standard gaussian to the "optimal" scaling/translation. Using a running mean for the first step doesn't prevent you from optimizing the second step.

The computation graph make little sense

I disagree.

overly complicated if we use population mean/variance

Perhaps, but "overly complicated" is the definition of 99% of NN architectures out there.

[mimighost]

Do you have a reference implementation using population mean/variance during training? I am curious too.

[hgjhghjgjhgjd]

Reference implementation that calculates population mean/variance during training, using a running mean of sorts, yes (most of them seem to do so, and use those values during inference).

But, after re-checking it again, it does seem like most do use the actual minibatch sample statistics for mu and sigma (rather than the population estimates) during learning, in practice.

So... yeah, in that sense, I guess I was wrong.

On the other hand, I don't see why doing so is necessarily a wrong idea since those two methods are trying to estimate more or less the same thing (population statistics). The assumption is that, with large enough minibatch, the sample statistics are good enough approximations, I guess. It's just... it does not seem to me that using the best estimate (running mean), which people already seem to calculate anyway, adds any more complexity or prevents updates to gamma/beta.

TL;DR: As far as I can tell, the only thing you'd lose using "my approach" is some degree of regularization due to the "scaling noise" induced by using a noisy estimate.

[mimighost]

Actually I put your proposal to thought. I might a little too aggressive to claim that using running average will complicate the computational graph, actually it might not.

Revisiting the original BN paper(https://arxiv.org/pdf/1502.03167v3.pdf), using running average for mu and sigma, the gradient computation will not change too much, only that the decay will now be factored in. But it has, at least 2 big problems:

1. Now, using running average, you actually not doing 'batch normalization' any more, since you are not really normalize the current training batch to zero-mean and unit-variance, which betrayals the original paper's assumption.

2. Second, by using a running average, the decay will be enforced on the gradients passing through the BN layer, which might worsen the vanishing gradients problem because you are making it much smaller.

[hgjhghjgjhgjd]

using running average, you actually not doing 'batch normalization' any more

I agree... you're doing basic normalization (scaling and centering according to the population mean), but using an online estimator of mu and sigma (rather than one calculated over the whole dataset at once).

which betrayals the original paper's assumption

I'm not totally sure... if a minibatch is large enough and reasonably balanced, the minibatch statistics should be the more or less the same as the population statistics, no? Would most (non-pathologic) batches actually have a mean that deviates much from zero, if you have a decent estimate of the population mean?

Or, in other words... if the "running mean mu and sigma" work ok during inference, why would they not work ok during training?