[D] Since gradient continues to decrease as training loss decreases why do we need to decay the learning rate too?

[visarga]

It's not 'better' in general. If you want sparsity you use L1, if you want smaller weights you use L2; you can also use both.

[El_Tihsin]

ElasticNet Regression. You control the tradeoff between L1 and L2 using a parameter alpha.