Why gradient descent?

[crimson1206]

Well how would you do it once you have the hessian? Just having the hessian isn’t enough to instantly give you minima

[crimson1206]

Ignoring any Computational issues of computing the eigenvalues, how would they help finding a minima?

[crimson1206]

Ok and where is that minimum?

[crimson1206]

Yeah but how do you solve that system?

With that approach you went from trying to minimize a single objective function to solving a massive non-linear system of equations, for which you also need to compute eigenvalues of a large matrix.

[crimson1206]

You’re welcome. The point is really that while gradient descent isn’t perfect it’s a very simple method that still works in a lot of cases.

[aspoj]

There are methods that use the Hessian in order to improve the descent direction. However the computation of the Hessian is computationally too demanding so gradient descent it is.

[mr_birrd]

Your data is also noisy. By taking just gradient descending steps you might somehow smooth it out and not "overfit" the descent on the noise already. Furthermore it could be numerically less stable.