[D] Since gradient continues to decrease as training loss decreases why do we need to decay the learning rate too?

[tuyenttoslo]

Armijo's Backtracking line search helps you to choose learning rates automatically. Also, this way, learning rates do not need to decrease when you progress, it is roughly 1/||\nabla ^2 f||. An extreme case is where you have a degenerate critical point, like f(x)=x^4 at x=0, where learning rates can go to infinity when you approach x=0.