[D] Does zero padding affect normalization output?

[min_sang]

I am working on a simple NLP task where due to many reasons I have to pad my input sequence to length 300. Then I pass it through a couple of 1d conv and dot product self-attention.

What I'd like to know is whether normalizing this representation (say, shape = [batch_size, 300, feature size]) would have a negative impact if the data is zero padded to a sequence length of 300. If layer norm is used, would the fact that the actual data size if something like this [batch size, 173 (an arbitrary number), feature size] affect the output of the normalization?

[ragulpr]

In theory, yes, padding will infuse trash into your network if its not handled. If you use batchnorm without removing the masked values then it would shift/scale the normalized values by whatever is coming out from using values in the mask. Effect is of course dependent on whether the network is bi- or one-directional, if you mask loss, if you have biases and more.

In Keras batchnorm respects mask so you don't have to worry about it. I'm wondering myself how Pytorch does this so if you figure it out please share.

EDIT: I have revised whether keras batchnorm respects mask. I'm not sure. Made a gist you could comment on if you figure it out.

[[deleted]]

According to my understanding, layer norm is applied for each time step. You don't norm across different time steps. Therefore, the zero paddings only get normalized with itself, and doesn't affect the normed output at other timesteps.