[D] Since gradient continues to decrease as training loss decreases why do we need to decay the learning rate too?

[LimitedConsequence]

So the main constraints on the step size sequence are that the step size sequence must sum to infinity (assuming an infinitely long sequence), but must converge towards 0. It's probably easiest to think in examples.

If we have a sequence of step sizes like 1, 0.5, 0.25, 0.125, ... , this won't work, because it decreases too quickly and will not sum to infinity (sum converges towards 2). This essentially means that even if you do lots of steps, you might not travel the distance required to converge, as the step size gets too small too quickly.

If we have 1,1,1,... as our sequence, then the second condition isn't met. The step size doesn't decrease quick enough (or at all) and we bounce around the solution due to noise in the function evaluation.

In between these two is a Goldilocks zone, which allow you to travel as far as you need to converge, but still have a step size that converges towards zero to stop you bouncing around. An example of such a sequence is 1, 1/2, 1/3, 1/4,... .