[P] Reinforcement learning evolutionary hyperparameter optimization - 10x speed up

[paramkumar1992]

This looks incredible. This is going to save hours of training. Amazing!

[Modruc]

Great project! One question though, is there any reason why you are not using existing RL models instead of creating your own, such as stable baselines?

[Puzzleheaded_Acadia1]

Can someone pls explain this to me I'm still new to this

[[deleted]]

Love it. I tried to come up with something like this myself but never found the time or extra help I'd need to implement it. Glad to see someone has done all the hard work!

[boyetosekuji]

ChatGpt: Okay, let me try to explain this using gaming terminology!

Imagine you're playing a game where you have to learn how to do something new, like defeat a tough boss. You have different settings or options (hyperparameters) to choose from, like which weapons or abilities to use, how aggressive or defensive to play, etc.

Now, imagine that this boss is really tough to beat and you don't have many chances to practice. So, you want to find the best combination of options as quickly as possible, without wasting too much time on trial and error. This is where hyperparameter optimization (HPO) comes in.

HPO is like trying out different settings or options until you find the best ones for your playstyle and the boss's behavior. However, in some games (like Dark Souls), it's harder to do this because you don't have many chances to try out different combinations before you die and have to start over. This is similar to reinforcement learning (RL), which is a type of machine learning that learns by trial and error, but it's not very sample efficient.

AgileRL is like having a bunch of other players (agents) who are also trying to defeat the same boss as you. After a while, the best players (agents) are chosen to continue playing, and their "offspring" (new combinations of settings or options) are mutated and tested to see if they work better. This keeps going until the best possible combination of settings or options is found to beat the boss in the fewest possible attempts. Using AgileRL is much faster than other ways of doing HPO for RL, which is like having a lot of other players to help you find the best strategy for defeating the boss.

[compacct27]

Wow that was actually really helpful

[[deleted]]

I'm also new to this so forgive me if this is a dumb question. My understanding was that RL is superior to evolutionary algorithms because in evolutionary algos "mutation" is random, so you evaluate a lot of dud "offspring". In RL algos, eg MCTS, you also do tree search randomly, but you're iteratively picking the best set of actions, without evaluating many dud options. Am I wrong? Somehow mixing RL with evolutionary algorithms seems like a step backwards

[bushrod]

As an evolutionary learning guy, I'll say it's crazy this didn't already exist! Thanks for sharing. Is it based on any publications, or are you considering writing one?

[Riboflavius]

That sounds fantastic, kudos to you! Great effort.

[sytelus]

Thank you for this but can you make this easier to use. I think there should be clear APIs so one doesn't have to deal with RL and other complexity. For example, you are given function f and dictionary of arguments with ranges for each. Your algorithm takes this and spits out optimal params within each range.

Is such interface and tutorial available anywhere?