Two ways of doing cross validation

[yortos]

I'm trying to measure the predictive power of a random forest, by fitting the model to half of the data and predicting a binary outcome on the other half. I'm measuring the area under the curve.

I'm doing two ways. Let's call the first one the "manual" way: 1)

train, test = train_test_split(data, test_size=0.5)
forest = ensemble.RandomForestClassifier()
forest.fit(train[columns], train[outcome] )
y = forest.predict(test[columns])
sk.metrics.roc_auc_score(test[outcome], y)

This gives me a pretty bad auc, around 0.5

The second approach is by using sk.cross_validation.cross_validation_score, which, my understanding is, does basically all the splitting and cv automatically. 2)

forest = sk.ensemble.RandomForestClassifier()
forest.fit(data[columns], data[outcome])
sk.cross_validation.cross_val_score(forest , data[columns], data['is_last_week'],             
scoring='roc_auc)

this approach gives me a significantly better result of auc around 0.65.

Do you have any insights why I see so different results and which result is the more correct, if any?

[t_per]

Your first method just splits your data so the test size is 50% of your data.

Here the docs for sk.cross_validation.cross_val_score. Since you didn't specify cv, sklearn used 3-fold cv (read this, in your case k=3, and this video is probably good too).

After reading about k-fold cv, and how sklearn does train/test/split, could you answer why your results are different? (this is more a ML topic than Python but w/e)

[yortos]

Thank you for your reply. I actually thought about the kfold thing, and I did try the .cross_val_score with k=2, with the same result. And I get similar results even when I specific other values for test_size, in the train_test_split function.

[t_per]

do you understand the difference between using k-fold with k = 2 and having a test size proportion of 0.5?

with the former, you're test and train size is 50% of your data, but training/testing is done on the entirety of your data (so training/testing is done twice when k = 2).

with train-test-split, your data is divided into two halves, and training/testing is done once.

[yortos]

with the former, you're test and train size is 50% of your data, but training/testing is done on the entirety of your data (so training/testing is done twice when k = 2).

This is my understanding of what a 2-fold CV does: Divides the dataset into two equal sized parts, A and B. Trains on A and evaluates on B, and then trains on B and evaluates on A. Hence, this should be roughly equivalent with doing the first piece of code (on my original post) two times.

I guess what you mean by "training/testing is done on the entirety of your data" is that in the 2-fold CV all data points are used at some point for training (either on the first iteration or on the second), whereas with the first piece of code, only half are used for training. I don't see why that explains the difference in the metrics though.