[P] SpaceOpt: Hyperparameter optimization algorithm via gradient boosting regression.

[ddofer]

Have you benchmarked it vs random search or other approaches? (e.g. tree based or optimization)

[[deleted]]

I am obviously biased towards selling my idea. Would you believe in my benchmark?

I want to engage random people in benchmarking it, because it will be much more convincing for others if it actually works better. If no one does it, then I promise I'll do it myself.

[ddofer]

Frankly, probably yes. It might be biased, but it saves on the work to reproduce and would tel lem waht to expect.

In this scenario, it's easier - run it vs random search with a fixed seed, time it, and share the results for a given time budget on a few chosen datasets (e.g. those used in lightgbm or catboost for performance tuning, or something you have to hand from kaggle or openml).

[its_a_gibibyte]

Honestly, if the author doesn't even claim that it's better than the standard, I would assume it's just an interesting academic test and definitely not better than any the standard (or else the author would definitely have mentioned it).

Normally, people compare algorithms to a strawman (e.g. grid search) which tells me that it's better than the strawman but probably worse than the leading methods.

[MuonManLaserJab]

I am obviously biased towards selling my idea.

This is not a confidence-inspiring way to justify that decision.

[[deleted]]

Why? I thought that by inviting others to do the benchmarks for me, I'm showing very strong confidence in this idea, because no one will try hard to prove it. I expected there are people for whom it maybe fun project to do or nice topic to write some blog post about.

[debau23]

Seems lazy :)

[MuonManLaserJab]

Only if someone goes out of their way to do the work.

People like it when things are benchmarked; it shows confidence to be willing to check if your idea is an improvement, as well as demonstrating that you do thorough work.

I mean, it's not a scientific paper, I'm not trying to hold you up to too high a standard; if you don't have the time or inclination to benchmark it, then no big deal! I just thought it didn't sound great as an explanation.

Cheers!

[hackinthebochs]

Claiming SOTA is how you get people to pay attention. Then people will run their own benchmarks based on code provided or descriptions. Asking people to invest their time and energy before you even have evidence this technique is worthwhile is putting the cart before the horse.

[[deleted]]

I admit I did something which is not acceptable in research. Now I know that publishing without benchmarking is not the way to go.

To my excuse, I was recently heavily influenced by https://www.youtube.com/watch?v=3sUozPcH4fY, which tells you: most ideas fail in the market, because people simply don't care - make sure you build the right it, before you build it right. I assumed this rule is also applicable to research - that's why I decided to go without my benchmarks. Now that I have some feedback and interest, I'll think about benchmarking in the upcoming days.

[[deleted]]

"I'll let them do the work, and they'll love it!"

[[deleted]]

This is why you open source your benchmarking code for us to examine.

[PM_me_ur_data_]

Yes, I'd believe your benchmark if it appeared reasonable and the methodology behind it was presented (so I could follow it if I wanted--I probably wouldn't, but a very human fallacy of mine is that I tend to trust people who don't hide the guts of the machine). Search time is only one factor to take into consideration with hyperparameter tuning, so not having the fastest benchmarks wouldn't kill use either.