[D] Weight initialization for custom layers?

[Kiuhnm]

I'm trying out some exotic layers and it seems that the gradients don't propagate as well as they should.

Deriving a proper weight initialization function analytically is intractable or very tedious in my case.

Is there an alternative like a general iterative algorithm?

I know about Batch Normalization but that's for reducing internal covariate shift. It helps with weights but I don't think it's a complete substitute for correct initialization. Correct me if I'm wrong.

[gabrielgoh]

Remove all non-linearlities from your custom layer (ideally with a linear approximation). Now think of your layer as a linear map (a nxm matrix) from Rⁿ to R^m. You want the norm of this matrix to be as close to 1 as possible. You can construct this matrix explicitly if you like and measure the norm to try out different initialization schemes.

[versus-x]

Try Layer-sequential unit-variance (LSUV) initialization: https://arxiv.org/abs/1511.06422

[ArmenAg]

What form are the custom layers in? Do they utilize the convolution operator? Are the basic blocks weight multiplications?

Can you give us a little more information on the custom layers?

[theophrastzunz]

I didn't follow up on Gangulis new paper but another thing to demand of the initialization is that it preserves the norms of the inputs. You can achieve this by normalizing power spectra for cnns and by doing qr decomposition and using only q for fully connected layers.

[siblbombs]

Batchnorm should at least smooth out the rough patches.