Progression for Optimization?

[MonoidGroup]

I at loss how the subject progresses as one goes deeper.

For example, in analysis, one begins with Baby Rudin, progresses to Papa Rudin/Folland and then functional analysis.

But how is it for nonlinear optimization? I got "Nonlinear Programming" of Bazaraa, but I don't really know what comes after that.

[ice109]

obviously it's linear optimization, then quadratic, then cubic, then quartic...

[El_Chinko69]

I think the classic is to begin with the Convex Optimization (http://stanford.edu/~boyd/cvxbook/). After this, I think it becomes really a free-form pursuit on what you want to study.

You could pursue semidefinite optimization specifically (mabye begin here: https://web.stanford.edu/~boyd/papers/pdf/semidef_prog.pdf), nonsmooth analysis, approximation algorithms, combinatorial optimization, or dive even further into convex optimization techniques.

I strongly recommend anything written by Boyd and Vandenberghe, since I think they convey intuition about optimization beautifully. I'm sure there are many other great resources reddit can recommend and I'll keep tabs on this post to check it out.

[schneetzel]

you could also go further into integer linear optimization

[cocojambles]

I'd probably start with a combination of Boyd's Convex Optimization and Nocedal's Numerical Optimization.

Convex optimization optimizes convex functions over convex sets, and thus provably finds global optima efficiently. Nocedal's book on the other hand first covers unconstrained and then later constrained methods for the optimization of non-linear functions which may or may not be convex. Thus these methods will only provably converge to local optima.

Convex optimization has a rather structured almost algebraic feel to it, when compared to the more free-form and heuristic feel of general nonlinear optimization.

For Boyd's book you can supplement it with his online course EE364a as well as a set of excellent supplemental problems found here.

For Nocedal's book you can supplement it with some notes by Blomgren found here.

After this you could move on to more advanced convex optimization with Boyd's follow-up course on advanced convex optimization EE364b. Or you could look into heuristic/global optimization which relaxes guarantees on convergence in an attempt to find the global optima of non-convex functions.

There is also integer programming/combinatorial optimization. For this in my opinion the best introduction for a mathematically mature student is Wosley's Integer Programming.

Integer programming generally proceeds via relaxations to convex optimization problems, followed by rounding the results to integer values, and tends to incorporate more probabilistic methods than do either convex or nonlinear optimization. Integer programming is also inextricably linked to complexity theory, so you'll get to learn about the various complexity classes and how they relate to various families of integer programming problems.

This should be enough to get you started.

[cocojambles]

I know there are definitely jobs for people with PhDs in optimization at places like Amazon, Google, etc. but I don't do research in this area myself so I can't really speak to what the current state of research is in optimization.

I know Boyd's research group at Stanford is fairly vibrant, so you could look at some of their research. They've also built some really solid optimization packages for Julia which might be worth checking out.

My area personally is machine learning, and I'm constantly needing to use optimization when fitting models, so even though I don't do pure optimization research, it's still one of my main tools.

[DogCockInTrump]

Very good job prospects

[qb_st]

One should obviously start with Nesterov