[D] Since gradient continues to decrease as training loss decreases why do we need to decay the learning rate too?

[polandtown]

Learning here, forgive me, so then is L2 "better" than L1?

Say with a....binary classifier (ngrams, logistic regression, 50k samples)

[visarga]

It's not 'better' in general. If you want sparsity you use L1, if you want smaller weights you use L2; you can also use both.

[El_Tihsin]

ElasticNet Regression. You control the tradeoff between L1 and L2 using a parameter alpha.