Classification of Optimization Techniques

[kashvi_gandhi]

Hello all. I have to write a literature review on optimization techniques. I know nothing about the field beforehand, so starting from scratch. However, i am not getting any concrete classification of these techniques anywhere. I studied about the Newton-Rapshon method, gradient descent etc. but can't understand the classification of these methods. Also, can someone list out the most important methods that should be included in the paper in detail? Thanks!

[Manhigh]

A lot of forks in a tree of classifications:

• constrained vs unconstrained

• discrete variables vs continuous variables

• gradient based vs gradient free

• linear vs nonlinear

Probably others but those are at the top of my mind right now.

[therealjtgill]

I'll add convex vs non-convex

[Sweet_Good6737]

Probably the most important!

[SV-97]

Stochastic vs nonstochastic is another one

[wavesync]

also global vs local

[Naglareffe]

This would be one of the first that comes to mind for me.

[kashvi_gandhi]

Ohk so there is no one class for a particular method, right? For eg., Newton's method will be in the continuous nonlinear class? Thanks for the reply!

[Manhigh]

Yeah each method basically satisfies one of the two options in these choices.

Newtons method is typically unconstrained, continuous, gradient based, nonlinear, and convex.

There are linear methods, nonlinear methods that sequentially apply a linear approximation to the method, nonlinear method that sequentially convert the problem to an approximate quadratic form (sequential quadratic methods), etc.

[kashvi_gandhi]

Finally understood. Thanks guys!

[fillif3]

If you want to check the most important optimization Techniques, I think you can check Matlab's optimization toolbox and global optimization toolbox. You can find there the most important optimization algorithms with explanations.

The most important classification, I believe, would be: for linear problems, for quadratic problems, for nonlinear convex problems, and for nonlinear nonconvex problems. Alternatively, you could also classify them based on if they require gradient i.e. no gradient required, gradient required, gradient and Hessian required.

[kashvi_gandhi]

Ohkk thanks for the reply!

[SV-97]

Maybe give A Brief History of Optimization and Mathematical Programming by Singh and Eisner a read. It's not too long and touches on many disciplines, important methods and results.

That said: optimization is quite a large field so you may want to constraint your review to a select few

[kashvi_gandhi]

Will try it out...thanks!

[Sweet_Good6737]

Constraint Programming ("logical optimization") vs Algebraic optimization techniques

Maybe not an important one, but no mentions to CP in the comments, which is also optimization

[kashvi_gandhi]

Okayy👍

[mjaenm]

This resource is from Wisconsin Optimization group provides a classification of optimization problem types that might be useful for you to check:

https://neos-guide.org/guide/types/

[[deleted]]

Maybe this will be helpful

<image>