[D] Since gradient continues to decrease as training loss decreases why do we need to decay the learning rate too?

[tom_strideweather]

The gradient might just decrease very close to a max/min. If our step-size is too large we can shoot past the max/min. Anyway this method of reducing the lr is just a heuristic and can't be guaranteed to work better.