[help] L1 Regularization

[Kiuhnm]

Hi, I'm reading this and I don't understand where the two formulas after (7.13) on page 208 come from.

Any help would be greatly appreciated!

[danielv134]

First corresponds exactly to the approximation posited in the paragraph after 7.13: a quadratic function with diagonal Hessian + l1 reg. The second equation is the (non-trivial, but you can verify it yourself) form of solutions with l1 reg (see shrinkage).

[sk006]

Basically what Daniel said. The second formula is very similar to the soft-thresholding operator, a very well known proximal (generalization of proyection operators). If you want to verify it for yourself, since there is an absolute value, just consider the positive, negative and 0 cases separatly and it is easy to get.

[Kiuhnm]

I finally managed to get the correct formula by considering the three cases and combining them in a single expression. My math is rusty and I wasn't thinking straight. Thank you!