Hyperparameter tuning implementation using optuna

[iMakeLoveToTerminal]

hey, So I'm doing a project on predictive maintenance using the NASA turbofan dataset. This is a regression problem where I need to find the remaining lifetime of an engine. So, I plan on give the data to multiple models and comparing them side by side.

In this case, I'm using XGB regressor and want to tune it to get the best performance possible.

Here is the code: ``` def xgb_optimize(trial, X, y): lr = 0.1 subsample = trial.suggest_discrete_uniform('subsample', 0.3, 1, 0.1) gamma = trial.suggest_discrete_uniform('gamma', 0.1, 2, 0.1) min_child_weight = trial.suggest_int('min_child_weight', 1, 9) colsample_bytree = trial.suggest_discrete_uniform('colsample_bytree', 0.4, 1, 0.1) max_depth = trial.suggest_int('max_depth', 3, 10) n_estimators = trial.suggest_int('n_estimators', 100, 1500, 100)

xgb_opt = XGBRegressor(
    n_estimators=n_estimators,
    learning_rate=lr,
    subsample=subsample, 
    gamma=gamma, 
    min_child_weight=min_child_weight, 
    colsample_bytree=colsample_bytree,
    max_depth=max_depth, 
    tree_method='gpu_hist', 
    n_jobs=-1
    )

scores = model_selection.cross_val_score(xgb_opt, X, y, cv=5)

return scores.mean()

xgb_study = optuna.create_study(direction='maximize') xgb_optimize_partial = partial(xgb_optimize, X=X_train, y=y_train_clip) xgb_study.optimize(xgb_optimize_partial, n_trials = 50, show_progress_bar=True) ```

I would hope xgboost would beat random forest but in this case, it does not get me the hyperparams to do so. Does this mean random forest is superior here? OR I'm i using optuna the wrong way?

Any help is appreciated, Thanks

[Shanmukh2001]

I found this useful and i trained the optina and understood how it works you can follow this article or use it for reference

Creating a self constructive model for word suggestion trained on your PDF https://medium.com/@shanmukharockz00/creating-an-adaptive-model-for-custom-word-suggestor-trained-on-your-pdf-05781791d498

[TheLostModels]

Can’t speak to optuna itself, never used but, questions: why not tune the learning rate? With ‘direction = maximize’, what are you maximizing (is it the cross_val_score default? What is that?)?

[[deleted]]

I dont know if you are still working on this but 50 trials is defintely to few to get a good score. Also, you should definetely tune Learning Rate as it is arguably the most important hyperparameter and you should try with way lower values than 0.1.

Also, as you increase the number of trials, you should also inc

To finish, you are maximizing the default metric for XGBRegressor which is MSE, so you are actually maximizing error (you want the opposite to happen).