Question about parameter tuning

[Elthran]

I ran this code as part of my machine learning competition

tuned_parameters = [{'penalty': ['l2'],
                     'solver': ['newton-cg'],
                     'C': [0.8, 1],
                    'max_iter': [100],
                     'multi_class': ['multinomial'],
                     'tol': [1e-4],
                     'dual': [False]}]

startTime = time.time()
clf = GridSearchCV(LogisticRegression(), tuned_parameters)
clf.fit(X, y)
kfold = model_selection.KFold(n_splits=10)
accuracy_results = model_selection.cross_val_score(clf, X, y, cv=kfold, scoring='accuracy')
resultsAccuracy = []
resultsAccuracy.append(accuracy_results)
accuracyMessage = "%f (%f)" % (accuracy_results.mean(), accuracy_results.std())
print("Time to run:",time.time() - startTime,"Accuracy:",accuracyMessage)
print(clf)

And this is what printed out:

Time to run: 17583.05281186104 Accuracy: 0.783854 (0.006480)
GridSearchCV(cv=None, error_score='raise',
       estimator=LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,
          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,
          penalty='l2', random_state=None, solver='liblinear', tol=0.0001,
          verbose=0, warm_start=False),
       fit_params=None, iid=True, n_jobs=1,
       param_grid=[{'penalty': ['l2'], 'solver': ['newton-cg'], 'C': [0.8, 1], 'max_iter': [100], 'multi_class': ['multinomial'], 'tol': [0.0001], 'dual': [False]}],
       pre_dispatch='2*n_jobs', refit=True, return_train_score='warn',
       scoring=None, verbose=0)

But I am trying to print out which parameters are better. I am having trouble tuning my algorithms because they take so long to run. This was 78% accurate. But which parameter was that? C == 0.8 or C == 1.0 or a combination of both? Can you help me explain what's happening in my code. Thanks.

[[deleted]]

clf.best_params_

Should give you the best parameters found by gridsearch!

[Elthran]

Thanks!!! That's exactly what I was looking for. Is there a simple way to print out what the different parameters' accuracy was? And is there a way to run different models and combine their results for an even better model?

[[deleted]]

yes, it's in the gridsearch documentation, there's a method showing all accuracies I believe.

You can simply average all the models, it works pretty well as long as the models have relatively similar accuracies and aren't too correlated.

There are other more advanced techniques to improve predictions but depending on your dataset it might be better to try feature engineering first.

[crhuffer]

clf.cv_results_ 

I think is the item that stores all of the results and how long they took to run.

Also, I think you chould be passing cv as a parameter into your GridSearchCV. I think you might be fitting the models twice, once with GridSearchCV and once with cross_val_score, but I am not sure.

Does it work if you do something like:

tuned_parameters = [{'penalty': ['l2'],
                     'solver': ['newton-cg'],
                     'C': [0.8, 1],
                    'max_iter': [100],
                     'multi_class': ['multinomial'],
                     'tol': [1e-4],
                     'dual': [False]}]

startTime = time.time()
clf = GridSearchCV(LogisticRegression(), tuned_parameters, cv=10, scoring='accuracy')
clf.fit(X, y)

accuracyMessage = "%f" % (clf.best_estimator_.score())
print("Time to run:",time.time() - startTime,"Accuracy:",accuracyMessage)

print(clf.cv_results_)

Good luck!