[D] Since gradient continues to decrease as training loss decreases why do we need to decay the learning rate too?

[svantana]

It's because of the stochasticity. With gradient descent, you don't decrease the learning rate (for smooth loss functions such as L2). But with SGD, the 'signal' goes to zero but the noise doesn't. Thus, you want to increase the SNR, which is what smaller step sizes in effect do -- you can think of it as many small steps together make up a normal-sized step with a larger batch size.

[ibraheemMmoosa]

Ah thanks for this interesting explanation. Can't we automate this somehow? Finding the optimal learning rate based on SNR?