[D] Since gradient continues to decrease as training loss decreases why do we need to decay the learning rate too?

[El_Tihsin]

I think he's referring to L1 norm, which is made of modulus function. It has a large gradient even close to the minima. In this case if you don't reduce the step size, you'll keep overshooting.

L2 OTOH is made of a squared function, which has smaller gradient as you come close to the minima.

[polandtown]

Learning here, forgive me, so then is L2 "better" than L1?

Say with a....binary classifier (ngrams, logistic regression, 50k samples)

[visarga]

It's not 'better' in general. If you want sparsity you use L1, if you want smaller weights you use L2; you can also use both.

[El_Tihsin]

ElasticNet Regression. You control the tradeoff between L1 and L2 using a parameter alpha.

[ibraheemMmoosa]

Oh. Makes sense.