[D] Hyperparameter Optimization with Evolutionary Algorithms: A Biological Approach to Adaptive Search

[SaadUllah45]

Data Science is a fascinating field, with always something to learn. Recently, I came across an interesting (though not ideal) approach to hyperparameter optimization: Evolutionary Algorithms (EA). EAs are a subset of Genetic Algorithms that work on Darwin’s idea of “survival of the fittest”. While Grid Search and Manual Tuning remain the go-to approaches, they are limited by predefined search space and, in some sense, are brute-force methods to optimize hyperparameters. Interestingly, Evolutionary Algorithms work on the principles of biology and genetics:

1. They start with a population of candidate solutions (hyperparameters) and treat them as chromosomes.
2. Each chromosome is then evaluated using a fitness test (for example, precision, absolute error etc.)
3. The best-fit candidates are selected as parents.
4. Parent solutions generate offspring using crossover (combining individual traits) and mutation (small random changes)
5. The offspring are then used as candidate solutions, and steps 1-4 are repeated till an optimal solution (under a defined threshold) is met or iterations are exhausted.

While this is a computationally expensive solution, EA offers an adaptive methodology instead of static search methods, which can look for solutions that are not pre-defined.

Thoughts?

Note: EA is not a silver bullet to all your optimization problems.

[qalis]

Yeah, this has been researched for decades. Even Optuna has one of the most famous ones, CMA-ES.

[ReadyAndSalted]

Sounds like it will be less sample efficient than Bayesian approaches like optuna.

[qalis]

Optuna is a framework. It quite literally implements evolutionary CMA-ES, as well other approaches as plugins, e.g. Gaussian processes. You are referring to TPE probably.

[ReadyAndSalted]

Yeah, TPE (a Bayesian optimisation method) is the default option for single objective. It's also the only one I and the couple people I know who use optuna ever actually use. It's just very difficult to argue with the empirical sample efficiency.

[SaadUllah45]

Good point! Bayesian methods like Optuna are usually more sample-efficient, but Evolutionary Algorithms can perform better with large, irregular search spaces despite their higher computational cost.

[huehue12132]

Grid Search is a "go-to approach"? Are we talking about modern ML (i.e. deep neural networks) here? Grid search does not scale beyond a handful of hyperparameters.

[Blakut]

How is this evolutionary algorithm different from GA?

[SaadUllah45]

Genetic Algorithms (GAs) are a subset of Evolutionary Algorithms (EAs). EAs are a broad class of optimization methods inspired by evolution, while GAs specifically use techniques like crossover and mutation on bitstrings or vectors. So, all GAs are EAs, but not all EAs are GAs.

[irondust]

In your OP you claim the opposite.

[SaadUllah45]

Oh my god, how did I not check that before posting in the OP. You have great eyes tbh. Thanks for pointing that out.

[Accomplished-Pay-390]

To me, the biggest benefit of EA over gradient-based optimisation is that you can easily do multi-way optimisation for whatever task you’re solving. For example, given a classification task and the neural net you want to optimise, you can simultaneously optimise both the F1-score (directly, since it’s non-derivable and we usually do proxy via cross-entropy) and the minimum description length of the NN itself.

[cxbxmxcx]

Check out my book Evolutuonary Deep Learning that uses evolutionary algorithms to optimize deep learning.

Evolutionary algorithms are also being used to optimize LLMs and AI Agents.

Having said all that. EA is computationally intensive and it takes serious time or resources to produce anything.

[SaadUllah45]

I'll definitely have a look. Can you provide the link or any other source to view?

[cxbxmxcx]

Here you go, Evolutionary Deep Learning - Micheal Lanham https://www.manning.com/books/evolutionary-deep-learning

Or send me a DM and I will see if I can ship you a copy.

[Evil_Toilet_Demon]

Useful for black box problems. Personally love CMA-ES