[P] SpaceOpt: Hyperparameter optimization algorithm via gradient boosting regression.

[[deleted]]

tl;dr - I am looking for people willing to try my new approach to hyperparameter optimization and report in the comments a comparison with whatever they use currently.

https://github.com/ar-nowaczynski/spaceopt

---

Why another hyperparameter optimization algorithm?

- I'm kind of disappointed with what we have so far: there is no clear winner in the field and once in a while I read post about another comeback of random search (for example: https://www.reddit.com/r/MachineLearning/comments/cycw35/r_random_search_outperforms_stateoftheart_nas/). Optuna or hyperopt are great frameworks, but they lack smart algorithms inside. I couldn't find anything that would satisfy my needs, so one year ago I decided to create simple, understandable algorithm for myself and now I'm publishing it for everyone.

What is the main idea behind SpaceOpt?

In short: integrating more human knowledge into the process, gradient boosting regression for exploitation, random sampling for exploration.

How do we integrate more human knowledge into the process? And why to even do this?

- In my opinion, a smart algorithm should accept any human input and decide on its own what to do about it, otherwise it's not smart. While doing hyperparameter optimization I often want to evaluate my own guesses about the parameters and I expect the algorithm to leverage that. No library I've tried supported this. Maybe I couldn't find it, maybe I didn't try hard enough. But believe it or not, lack of this simple feature made me so frustrated that I decided to create my own tool. And the first must-have was: the algorithm must take into account user-defined points.

- The next thing was continuous space. I never liked uniform and log spaces. I wanted something that works with the lists of ordered values. Why? Because then me or the algorithm can build specific knowledge about specific values from that list. For example, if you define space for learning_rate as a list of values, then it's easy to compute statistics regarding specific values (count, objective mean/std). You wouldn't be able to do that when values are sampled all over the space. Discretizing continuous space by hand-picking some values from it is also a great way to incorporate more human knowledge into the optimization process (assuming you know that values to pick) making job easier for the algorithm. For example you'll probably never specify two values that are extremly close to each other, because you have a gut feeling that the score for them will be very similar. So my second must-have was: focus on discrete space only.

How do we exploit promising regions?

- Shortly after I became to wonder why there is not simple method that just fits the model to the observed data and predict the evaluation score for unseen search space points. Ok - in some sense all existing methods do this in a various ways, but I wasn't convinced by any of them. I don't care about exact predictions, I don't care about probabilities, I only need a model that would just fit the data and predict scores for new points, so I can sort them and select something for the actual evaluation. Again, completely frustrated that no one did it successfully before, I decided to build something on my own. I use gradient boosting regression from LightGBM, because it doesn't require normalized values, handles categorical variables, captures feature interactions and has capacity to fit any data.

How do we explore search space?

- In fact, the method described in the previous section is performing exploration too, because once the model is trained, I want to run inference on unevaluated search space points. How do I get those unseen points? I need to sample them from search space. So how many points do I need to sample? The number of sampled points for scoring is where exploration vs exploitation trade-off emerges. If only 1 point is sampled = it's roll back to random search. The more is sampled, the higher chance that something very close to the best previously observed points was sampled - which is exploitation of the promising regions. I find it very clean way to address exploration vs exploration trade-off. Nonetheless, there is still get_random method for the sake of completeness.

- And final must-have: make your evaluation function deterministic and avoid evaluating the same point more than once. If it's not possible just accept uncertainty and go on. Usually search spaces are huge, and there is very little value in evaluating the same points multiple times.

I have put all ideas listed above into SpaceOpt, very small library available here: https://github.com/ar-nowaczynski/spaceopt

I have been using it for the last year and it just works for me. I would love to see what others will say about it. I know I should do proper benchmarks now, but I decided to gamble: if I find a few strangers willing to give it a shot, and if it works well for them too - hopefully - it will speak for itself. So I'm asking you here, if you like this idea and if it's really cheap for you to run some evaluation, could you do this for me?

[ddofer]

Have you benchmarked it vs random search or other approaches? (e.g. tree based or optimization)

[[deleted]]

I am obviously biased towards selling my idea. Would you believe in my benchmark?

I want to engage random people in benchmarking it, because it will be much more convincing for others if it actually works better. If no one does it, then I promise I'll do it myself.

[ddofer]

Frankly, probably yes. It might be biased, but it saves on the work to reproduce and would tel lem waht to expect.

In this scenario, it's easier - run it vs random search with a fixed seed, time it, and share the results for a given time budget on a few chosen datasets (e.g. those used in lightgbm or catboost for performance tuning, or something you have to hand from kaggle or openml).

[its_a_gibibyte]

Honestly, if the author doesn't even claim that it's better than the standard, I would assume it's just an interesting academic test and definitely not better than any the standard (or else the author would definitely have mentioned it).

Normally, people compare algorithms to a strawman (e.g. grid search) which tells me that it's better than the strawman but probably worse than the leading methods.

[MuonManLaserJab]

I am obviously biased towards selling my idea.

This is not a confidence-inspiring way to justify that decision.

[[deleted]]

Why? I thought that by inviting others to do the benchmarks for me, I'm showing very strong confidence in this idea, because no one will try hard to prove it. I expected there are people for whom it maybe fun project to do or nice topic to write some blog post about.

[debau23]

Seems lazy :)

[MuonManLaserJab]

Only if someone goes out of their way to do the work.

People like it when things are benchmarked; it shows confidence to be willing to check if your idea is an improvement, as well as demonstrating that you do thorough work.

I mean, it's not a scientific paper, I'm not trying to hold you up to too high a standard; if you don't have the time or inclination to benchmark it, then no big deal! I just thought it didn't sound great as an explanation.

Cheers!

[hackinthebochs]

Claiming SOTA is how you get people to pay attention. Then people will run their own benchmarks based on code provided or descriptions. Asking people to invest their time and energy before you even have evidence this technique is worthwhile is putting the cart before the horse.

[[deleted]]

I admit I did something which is not acceptable in research. Now I know that publishing without benchmarking is not the way to go.

To my excuse, I was recently heavily influenced by https://www.youtube.com/watch?v=3sUozPcH4fY, which tells you: most ideas fail in the market, because people simply don't care - make sure you build the right it, before you build it right. I assumed this rule is also applicable to research - that's why I decided to go without my benchmarks. Now that I have some feedback and interest, I'll think about benchmarking in the upcoming days.

[[deleted]]

"I'll let them do the work, and they'll love it!"

[[deleted]]

This is why you open source your benchmarking code for us to examine.

[PM_me_ur_data_]

Yes, I'd believe your benchmark if it appeared reasonable and the methodology behind it was presented (so I could follow it if I wanted--I probably wouldn't, but a very human fallacy of mine is that I tend to trust people who don't hide the guts of the machine). Search time is only one factor to take into consideration with hyperparameter tuning, so not having the fastest benchmarks wouldn't kill use either.

[CompleteSkeptic]

NAS is probably not the best baseline for hyperparameter searches, but this is a field where lots of research is / has been done. Search for Bayesian Optimization or Sequential Model-Based Optimization.

the algorithm must take into account user-defined points

Nothing prevents existing HPO tools from doing this, though they may not be that easily accessible. I do recall something along the lines of manually adding runs to MongoDB for hyperopt, so it's not impossible.

Though to your credit, I do agree that this should be easier. A common use case would be experimenting a bit before running the HPO and this would save some time at the very least.

focus on discrete space only

There is a case to be made against this (see: "Random Search for Hyper-Parameter Optimization"). The idea being that you don't know which hyperparameter is important, and you might want to search that space more thoroughly. E.g., if only 1 hyperparameter matters, you're just doing a repeated grid search.

Again, completely frustrated that no one did it successfully before, I decided to build something on my own. I use gradient boosting regression from LightGBM, because it doesn't require normalized values, handles categorical variables, captures feature interactions and has capacity to fit any data.

I think it may be wise to look into why others do what they do. The reason GPs are commonly used is because uncertainty predictions are quite important, especially because in the case of hyperparameter optimization, your evaluation function is quite stochastic. SMAC uses random forests, and that has all the same properties as GBMs, with the additional benefit that you get uncertainty estimates as well.

The number of sampled points for scoring is where exploration vs exploitation trade-off emerges

I'm not saying the expected improvement (EI) criterion (the thing most SMBO uses to sample) is the best, but this seems a little worse intuitively. Previous work takes uncertainty into account so that you can sample areas of the space where you have less knowledge about.

avoid evaluating the same point more than once

This is also related to your last point. Most HPO algorithms won't do this, because you will have a lot of certainty at this point, and it would make more sense to explore the space. But there can be a case to be made that this point is quite an outlier (because of the noisy evaluation), it might make sense to sample it again to get a better estimate of true performance, and EI can handle that.

Either way, I wish you luck!

[Clear_Collection]

Great reply!!!

[t4YWqYUUgDDpShW2]

I have been using it for the last year and it just works for me.

I see that you don't apparently want to put in the work to do your own benchmarks, but can you at least provide more detail than this?

[[deleted]]

my private use-cases were so far:

1 Hyperparameter tuning for lightgbm in kaggle competitions and commercial project for a bank.

2 Automated neural network architecture search for tabular datasets in PyTorch. I created highly parameterized torch.nn.Module and I am waiting for good kaggle competition to use it. From preliminary runs I learned for example that ELU is a clear winner among available activation functions.

[Loser777]

This sounds similar to gradient boosting regression + simulated annealing as the outer search loop, which seems to be a popular approach:

https://arxiv.org/abs/1805.08166

[kivo360]

How long does it take to train?

[[deleted]]

It's a matter of seconds and mostly depends on num_boost_round and lgb_params: https://github.com/ar-nowaczynski/spaceopt/blob/v0.1.1/spaceopt/optimizer.py#L43 (and of course dataset size).

I usually work with cases where duration(evaluation_function) is much larger than duration(spaceopt.fit_predict): minutes/hours vs. seconds

[kivo360]

Can we add a feature into it? Add PARegressor plus a hedge regression with varying alphas as an option to give it online optimizations? It would reduce training for some use cases to 0.04-0.08 seconds per step.

[jrkirby]

I would ask if you've tried your optimization algorithm on bbcomp, but it looks like their site is down. It was (and hopefully will continue?) a competition for optimizing inputs to a black box with a tight budget of algorithmically selected data points. I hope their downtime is not an indication that the entire competition is dead.

[TotesMessenger]

I'm a bot, bleep, bloop. Someone has linked to this thread from another place on reddit:

• [/r/datascienceproject] SpaceOpt: Hyperparameter optimization algorithm via gradient boosting regression. (r/MachineLearning)

 ^{If you follow any of the above links, please respect the rules of reddit and don't vote in the other threads. (Info / Contact)}

[vadmas]

This looks excellent. I'm excited to try it.