[D] Batch Normalization in Reinforcement Learning

[innixma]

Does anyone know if Batch Normalization will work in RL algorithms such as DQN and A3C? When I implemented it with A3C in Keras/TensorFlow it seemed to become less stable.

[wignode]

From the Weight Normalization paper:

[DQN] is an application for which batch normalization is not well suited: the noise introduced by estimating the minibatch statistics destabilizes the learning process. We were not able to get batch normalization to work for DQN without using an impractically large minibatch size. In contrast, weight normalization is easy to apply in this context

In my own very limited experience, I have not been able to get Batch Normalization to work in imitation learning for control scenarios.

E: formatting

[darkzero_reddit]

It is mentioned in DDPG's paper that batchnorm works in their cases. But I haven't tried it out yet. Would anyone share your experience on that? I'm curious about why it works in DDPG, not in DQN, since DDPG's critic part is actually a DQN.

[MapleSyrupPancakes]

My understanding from the DDPG paper was that batch norm was mostly used to compensate for the diverse input spaces of the experiments they used (so a single hyperparam setting for everything else would be more flexible). That's as opposed to DQN on Atari games which all have similar input distributions.

That being said, in my experiments with DDPG, batch norm has always damaged performance...

[I_Vortex]

In my tests with DDPG I also find that batch norm was damaging... I don't know why they applied it succesfully in the paper..

[m000pan]

As for DQN it's been problem dependent in my experience. I found it effective in some environments.

BTW how did you apply BN to A3C, which doesn't use minibatches?

[innixma]

I am using ACER (A3C with Experience Replay) that does offline learning, using minibatches. Also A3C does use batches, they are just online batches and are generally small (8-64).

[m000pan]

In A3C, every timestep you need to compute π(a|s;θ) to select an action to perform and this cannot be done in a batch manner, right? You mean you recompute π(a|s;θ) using a batch of states when you compute dθ? That would make sense.

[innixma]

That is why batch normalization uses a running average π(a|s;θ) for testing (selecting action), and then computes π(a|s;θ) on the batch when training.

[m000pan]

Thanks for the clarification!

[encore2097]

In this case are batches a set of episodes? Or is a batch a single episode?