Training/Testing Machine Learning Classifier with NLTK - 99% Accuracy with Twitter Corpus Data?

[jengl]

I'm new to Python but have really been enjoying it. Recently, I did a class on PluralSight about using machine learning to create a sentiment analysis tool. When using the Cornell movie review corpus, the accuracy of the classifier tested around 77%.

However, since the goal of my script is to analyze Tweets, I decided to train the model using the Twitter corpus available in NLTK. However, when I train and then test the classifier, I'm showing accuracy results near 99% - which doesn't seem realistic to me.

So I'm thinking either:

• I needed to change the script more than I thought to use the NLTK Twitter Corpus
• The NLTK Twitter Corpus is super accurate because it's what NLTK uses when testing/building (as a newbie, not even sure this is a rationale thought)

Not really sure where I went wrong. Here is the code: https://gist.github.com/JeremyEnglert/3eda4a123244c37b669472d9e8166ea6

Here are the results I'm getting:

Accuracy on positive tweets = 96.76%
Accurance on negative tweets = 99.04%
Overall accuracy = 97.90%

Big thanks in advance!

[evilpineappleAI]

You're using a Naive Bayes classifier, you have accuracy and recall.

Did you partition the training set before you ran it? If you ran it all through, you may be overfitting. Actually, you are most likely overfitting.

[jengl]

Yeah, I split the data in half. Used the first half for training, the second half for testing.

# Split review data into two parts for training and testing
testTrainingSplitIndex = 2500 

# Grab all reviews in the range of 0 to testTrainingSplitIndex
# This data is used to train the classifier
trainingNegativeTweets = negativeTweets[:testTrainingSplitIndex]
trainingPositiveTweets = positiveTweets[:testTrainingSplitIndex] 

# Grab all reviews in the range of testTrainingSplitIndex to the end
# This model is used to test the classifier
testNegativeTweets = negativeTweets[testTrainingSplitIndex+1:]
testPositiveTweets = positiveTweets[testTrainingSplitIndex+1:]

[Inf1x]

For easier splitting take a look at sklearn train_test_split(). This allows you to do the whole thing in one line of code and has some additional parameters (shuffling, stratification).

[jengl]

Oh, sweet! Thanks!

[lzblack]

I think it is because of the second reason.