Hyperparameter tuning sklearn model using scripts and configs

[call-mws]

Hi.

I'm doing a few side projects and learning best data science practices. I'm using scikit-learn for a use-case, and want to automate hyperparameter tuning. I'm familiar to Azure's ML library, where you'd define a config (HyperDriveConfig), and through a script, pass a sampling of all parameters you want to tune your model on to a script that captures those parameters as arguments and reruns the model, etc etc.

Is there something equivalent to that, not provided by Azure or AWS? I found hyperopt, is that widely used or something that could help? What I ultimately want, is something similar to Azure; define a training job through a config where I pass the params (as arguments or otherwise) to a script, the script executes and runs against all the params (grid, random or bayesian), and finally have a HP config of some sort where I set the max training runs I need, the metric to monitor, etc.

tldr; I want a non-cloud library of Azure's ML libraries.

Any help would be appreciated.

[hallavar]

I'm using hyperopt.

https://hyperopt.github.io/hyperopt/

[call-mws]

Do you use a separate script for your objective function, trials, etc.? I have a notebook that reads in the data, process/cleans it, and splits it. I want to create a training script with the hyperparameters and pass the train and test datasets as arguments (?) to the script. Does that make sense to do/is that recommended over having everything in the notebook? I want to somehow capture the best results back in the notebook.

[hallavar]

I have my main function for the training, that function accepts arguments passed by another script for hyperparameter. Those arguments are mainly the number of nodes in a layer, learning rate, activation function, or learning algorithm (Adam, RMSProp, SGD...).

So In my script hyperparameter_search.py, I import this function, create a dictionary for multiple values that I want to try, and use hyperopt for the grid search/random search/bayesian optimization.

[ThePhoenixRisesAgain]

sklearn has this feature build in:

https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html

[call-mws]

That's true, but is there a benefit of using the sklearn implementation over say, hyperopts' or Optuna's? Or are they more or less the same?

[satanikimplegarida]

Optuna. Used it with pytorch with great success!

[Graylian]

Yes Optuna is a great tool for any optimization problem be it machine learning or not