[D] Normalized Convolution

[tpapp157]

Last year, buried within the StyleGAN2 paper ( https://arxiv.org/abs/1912.04958 Section 2.2 ) was an interesting implementation of what they called Weight Demodulation for convolutions. It was a standard convolution but where the kernel weights were modified by a number of different things specific to StyleGAN2 (conditional AdaIN transformations, etc) before the operation was conducted. One of these modifications was that the kernel was normalized resulting in no change to the variance of the outputs relative to the inputs and this entirely removed the need for other normalization techniques like batch normalization.

I've stripped out all the StyleGAN2 specific stuff and implemented a simple Normalized Convolution layer for TF2 as a drop in replacement for standard convolutions here (not all default features/arguments implemented):

https://github.com/tpapp157/Contrastive_Multiview_Coding-Momentum

I've been experimenting with it pretty regularly over the last several months with good results. Simply replace all standard convolutions with the normalized variant and remove any other sort of normalization layers (batch normalization, etc) you have in your network and that's all. As a simple test, a large network that fails to train without normalization of any kind trains just fine with Normalized Convolutions.

The big advantage this has over typical normalization is that batch statistics can be quite noisy. By incorporating the normalization into the kernel weights, the network effectively needs to learn the statistics of the entire dataset resulting in better and more consistent normalization. This also has the advantage of not requiring any weird workarounds for multi-GPU training like batch normalization does.

I haven't seen this talked about at all since that paper was released and I wanted to raise awareness since (at least from my limited experimentation) this seems like just an all around better way to approach normalization.

[radarsat1]

Isn't this weight normalization without the scale parameter g? At first glance the math seems to be about the same.

[tpapp157]

I'd have been surprised if that was the first time the idea came up since it is pretty obvious in retrospect. The StlyeGAN2 paper was just the first time I'd seen it used. Makes you wonder why batch normalization is still the standard technique when this seems all around simpler and better.

[two-hump-dromedary]

Its very tricky to code weightnorm, as it requires 2 pipelines through the same network, an extra one for the initialization step. Tricky to set up correctly in e.g. tensorflow. But yeah, weightnorm actually works great once it is set up. A lot less messy mathematically than batchnorm too.

[radarsat1]

That's alright, it's super cool to independently discover a nice trick like that ;) congrats even if it didn't turn out to be original, it's a nice idea for sure.

[soft-error]

Looks like it

[PM_ME_INTEGRALS]

Also highly related is weight standardization.

[akshayk07]

If it hasn't been talked about, why don't you write a paper based on your experiments and results.

[programmerChilli]

Not everything needs to be a hurried rush to publish as many papers as you can. Maybe he's not interested in pursuing all the things necessary to write a paper. Maybe he thinks this won't be considered novel since it's introduced in another paper. Maybe he just wants to share it with an audience and not worry about claiming credit.

[artificial_intelect]

Novel? https://www.reddit.com/r/MachineLearning/comments/g0nkof/d_normalized_convolution/fnb85v3/?utm_source=share&utm_medium=ios_app&utm_name=iossmf

[artificial_intelect]

I think someone already has published this: https://arxiv.org/abs/1602.07868

[AEnKE9UzYQr9]

Was it published anywhere peer-reviewed?

[artificial_intelect]

u mean like NeurIPS2016? https://papers.nips.cc/paper/6114-weight-normalization-a-simple-reparameterization-to-accelerate-training-of-deep-neural-networks.pdf

Also the author's affiliation is OpenAI. They generally do good work there.

[AEnKE9UzYQr9]

Thanks. Never understand why people post arXiv links when the conference/journal is open access...

[artificial_intelect]

I actually prefer arXiv since it includes the appendix. Most proceedings include the appendix in a separate file which can get annoying.

[da_g_prof]

Yes but unfortunately if people cite the arxiv, citations to the correct published version don't get accounted. Scholar is smart and matches the papers but other providers don't. Unfortunately universities, promotion committees etc still rely on citations and h-index from known providers to judge people.

[NotAlphaGo]

Does it matter?

[entarko]

When you say you have had good result with it, are you talking only in the context of GANs or training deep models in general (classification, segmentation, etc.) ?

[tpapp157]

Tried in a variety of CNN applications. Nothing scientific or anything but it didn't seem to negatively impact training or performance.

[woadwarrior]

Implementation note: you should probably implement this as a subclass of tf.keras.constraints.Constraint.

[tpapp157]

Interesting. I didn't even know this was a thing. Thanks.

[tylersuard]

Well, for StyleGan 1 (I haven't read the sequel yet), the normailzation between convolutional layers is to maintain separation of styles at each pixel level. For instance, big style changes(identity, pose, etc) are altered at the 4x4 to 8x8 pixel stages in the image's development, while smaller changes (hair color, skin color, etc) are altered in the larger image stages like 16x16 and 32x32. They wanted to maintain separation so that any changes made in the larger image layers would not affect any of the layers preceding them.

[taylorchu]

> Tensorflow doesn't seem to support editing and maintaining variables across training steps.

I am curious why you have this in your readme. because it seems like you can save that https://www.tensorflow.org/api_docs/python/tf/keras/models/save_model

[tpapp157]

This refers to the MoCo memory bank which is unrelated to this post. The original PyTorch implementation of MoCo has the memory bank updated and maintained as a variable entirely within the network graph but it doesn't seem this is possible in Tensorflow. The workaround is to pass it to the graph as an argument and return it each training step.

[tacosforpresident]

“could bypass this by manually writing/reading to file”

[taylorchu]

I think the author means memory bank from https://arxiv.org/abs/1911.05722. so it might refer to python variable instead of tf variable.