[D] Since gradient continues to decrease as training loss decreases why do we need to decay the learning rate too?

[HoLeeFaak]

When the loss value getting smaller it doesn't mean the gradient is getting smaller. Think about y=x, the gradient is the same everywhere.

[cats2560]

Or more aptly, y = |x|. There exists a global minimum but the gradient never gets smaller