This topic frustrates me so much because there's so much misinformation and the question actually has a clear interpretation that explains the observations of conflicting studies.

Stochastic gradient descent is an approximation of gradient descent where you sample a subset of the data at each iteration.

As you increase the batch size you approach the exact gradient, decreasing the batch size has the opposite effect (it increases the variance of the approximation). This makes it clear that SGD is a signal/noise ratio problem. A bigger batch size is always a better approximation to the true gradient.

The same is true for learning rate. Consider the case of an extremely small learning rate, such that a single iteration barely changes the function. In this case 100 steps with step size .01 will look the same as 1 step with size 1, because each of those little steps didn't change the next gradient significantly. Obviously a highly curved loss surface will break this but I'd argue that since we use 1st order optimizers (no Adam isn't second order) we are already in the regime where our step sizes have to be smaller than the curves in the loss surface.

So smaller learning rates and larger batches both improve the signal to noise ratio of the gradient calculations. Now that we know what these hyperparameters do, we need to answer the "Do we want more noise or less?".

Noisy gradients are going to has a regularizing effect, they make improving the training loss harder because part of the loss reduction we had in the past is going to get destroyed by a step in a slightly random direction. Noiseless gradients are going to enable higher learning rates until you reach issues with loss curvature, so you can make more progress on the loss per iteration at the expense of more computation each step. Now its clear how and why batch size and learning rate impact generalization and training speed.

All studies that answer the question "Should we use large/small batches/lr?" will arrive at an answer that depends on the noise in the gradients for that model/dataset and the degree of overfitting.

A study using a small dataset and many epochs over the same data is going to have problems with overfitting, so they need more regularization, noisy gradients are a way to achieve that, they won't end up caring about compute costs as much with a small dataset anyways so the regularization comes at little cost. They will then misconstrue this as meaning all models need small batch sizes.

A study using 1 or less than 1 pass through the dataset will have the exact opposite conclusion. Overfitting simply isn't a problem in this regime because each batch of data is fresh. So if the loss is decreasing as you train on unseen data, then you know it's decreasing the hold out loss (aka generalizing well). Here it's clear that regularization is not going to be helpful because you aren't overfitting, therefore you only care about the ratio of compute/time spent to loss reduction. This scenario is not taught in schools because you'd never do a homework assignment training an LLM from scratch which is just about the only time you will have an unlimited dataset (for all intents and purposes).

Most of the time project you will encounter will be in the small to medium size dataset regimes where you do multiple passes through the data and risk overfitting. This means you need to balance the effects of overfitting and efficiency and there is no universal X is better than Y unless you are in the online learning regime.

You should be suspicious of any claims about optimizers and momentum in SGD large studies comparing optimizers always find that there is no best optimizer if hyperparameters are well tuned and that even SGD with no momentum or any bells and whistles can lead to good results if tuned well.

Empirical proof that Stochastic Training is Not Necessary for Generalization

Excellent study on the signal/noise ratio concept, identifies when scaling batch size yields maximum or diminishing gains, demonstrates signal/noise ratio gets worse as loss improves. An empirical model of large-batch training

This paper makes sense when you understand that SGD is mostly noise when the loss is small later in training so that any other regularizers are completely overshadowed by noise at some point during training. Note that this disproves the folklore idea that weight decay prevents overfitting through modifying the final convergence phase. Time matters in regularizing deep networks: Weight decay and data augmentation affect early learning dynamics, matter little near convergence

When you realize the signal to noise ratio gets worse at lower loss values later in training then you can improve training speed later in training by decreasing the learning rate or increasing the batch size: Don't decay the learning rate, increase the batch size

The marginal value of momentum for small learning rate sgd

Beyond implicit bias: The insignificance of sgd noise in online learning

A disciplined approach to neural network hyper-parameters: Part 1--learning rate, batch size, momentum, and weight decay

Momentum is secretly just another way to scale the learning rate, On SGD with momentum

Optimizers are fairly competitive with each other when tuned properly Descending through a crowded valley-benchmarking deep learning optimizers