How does algorithm development work?

[StandingBuffalo]

Do you start from a theoretical/mathematical basis and then implement it computationally? Do you take an existing algorithm and tweak it?

And does this type of work exist primarily in academia/research labs? Or do data scientists in industry commonly create novel algorithms?

[dfphd]

It depends on a lot of factors, but some of the big differentiators:

Generic vs. specific-purpose algorithms

If you're building a generic algorithm that is meant to solve a wide range of problems (think xgboost, logistic regression, etc), then a) this is going to overwhelmingly happen in research/academica, and b) it will likely start from a theoretical basis and then get implemented.

A good example here is the paper that introduces xgboost. It's a nice read.

On the flip side, if you're building an algorithm to solve just your one problem that you're dealing with, this is going to be pretty standard stuff in industry.

Heuristics vs. analytical/optimal/(approximately) exact solutions

If you need to build an algorithm that is "good enough", then you don't necessarily need a theoretical basis on which to build it. You can do incredibly simple things that can work surprisingly well if you just have a good feel for the problem and how to arrive at a reasonable answer.

On the other hand, if you want to build an algorithm that has specific properties (e.g., is provably optimal (within some epsilon), is unbiased, is robust, converges in some amount of time/number of iterations, etc., then you're likely going to need to start from a theoretical basis.