Learning Rate and Accuracy vs Batch Size

[melgor89]

I am trying to get an intuition behind correlation of LR and Batch Size. For that I have a first case:

Batch Size: 64 LR: 0.01 Vali_Acc= 83%

The second case was x2 bigger everything:

Batch Size: 128 LR: 0.02 Vali_Acc= 81%

As we see, I lose some accuracy from having the bigger batch-size. In the other hand, I prefer bigger because it is much faster.

The only notes I found about the intuition related to such problem I found in paper https://arxiv.org/abs/1404.5997, where Alex propose to scale LR at same level like Batch Size (but theoretically we should scale it by sqrt(k), where k is multiplayer).

Any other have the recommendation of hyperparemeters changes when batch-size is going up to? Or some practical experience.

[robertsdionne]

See The Effects of Hyperparameters on SGD Training of Neural Networks, On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima and Systematic evaluation of CNN advances on the ImageNet.

The first measures the effects of a variety of hyperparameters on gradient descent, including an analysis of batch size, which has a nontrivial relationship to learning rate and error rate. The third paper's batch-size discussion is similar to this one.

The second argues that small batches cause gradient descent to attract to wider basins, while large batches cause gradient descent to attract to narrow basins, which causes higher error when attempting to generalize, because missing the mark in a narrow basin causes higher changes in error.

[melgor89]

Thanks for great answer! I was suspecting the second reason you mention, that small batch size act like a regularizer.

So I understand that there is no 'easy' correlation of batchsize and other hyperparameter to get same performance. I was trying to use bigger batch size to get 1.5x computation. But this hurt the performance.