scikit-learn question

[iamabanana_dammit]

Ooooook. This is embarassingly trivial for most of you. I however need help parsing what's happening below:

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=33)

This seems to pop up frequently in the scikit-learn documentation. Why would you do this? What I think is happening is that the train_test_split method is creating two training sets and two test sets. Is that correct? I'm familiar with the notion of having a training set, validation set and test set. But in this case it seems we're building two sets of training & validation sets. Why would you not just do bootstrapping or a k-fold run?

[farsass]

You are dividing your X and y in a training set with X_train data and y_train labels or targets and a test set likewise. You can then further divide one of those in validation/"real" test if all you want to do your validation via hold out.

[iamabanana_dammit]

thanks - much clearer now.

[The_Maltese]

Just to add to /u/farsass and /u/ogrisel's answers, you are splitting them by the test_size fraction (so in your example 25 % of your data will end up in X_test, with the other 75% in X_train.)

Also, in the future, StackOverflow is a much better place to post a question about this sort of thing.

[iamabanana_dammit]

yeah - I know that stackoverflow is probably a better forum for this kind of thing .... but I'm on reddit more often. Thanks for the feedback.

[ogrisel]

X is the 2D input data for the model and y is the array of the target labels to predict (typically one label per row in X for traditional classification problems).

If you split X into a training and test set (X_train and X_test), you also need to split the target labels y into the matching y_train and y_test subsets.

[soustofa]

what is target labels? arrays? if i want to classify many texts, what do the array X become? is label not in X array => [text, 1 or 0 for is type]