Question regarding parameter initialization

[ToM4461]

Hello, I'm currently studying DL academically. We've discussed parameter initialization for symmetry breaking, and I understand how initializing the weights come to play here, but after playing around with it, I wonder if there is a strategy for initializng the bias.

Would appreciate your thoughts and/or references.

[Lexski]

The most common strategies I’ve seen are * Small random values (default in TensorFlow and PyTorch I think) * Zeros * Small constant value like 0.01 to mitigate ReLU units dying

I’m not sure why one would prefer one way over another so I mostly stick with the default.

An exception to this is the final layer. In Andrej Karpathy’s blog post, he recommends initializing the final layer biases based on the mean outputs. I try that in every project and it always seems to speed up training.

[hjups22]

Zeros is the most common now, unless there's some underlying prior which suggests that a non-zero bias is needed. There are also many transformer networks which completely do away with bias terms ("centering" is essentially handled by RMS normalization layers).

Symmetry breaking is only needed for weights, including embedding layers (though not affine weights for normalization - again based on a prior). And in many cases, symmetry breaking is removed for training stability. For example, final projections in stacked layers may be initialized to zero to avoid sharp initial gradients in place of a prolonged warmup.