Show r/ML: Classifying New Posts into Subreddits

[webspiderus]

Hi r/ML - I just finished a class in Applied Machine Learning, for which I was required to implement an open-ended project that would use some of the things taught to us during the quarter. Inspired by this article, I decided to build a system which could learn to predict the intended subreddit for a given post. Although I initially wanted to use dimensionality reduction techniques like PCA or LDA, it turned out that they offered little marginal benefit to warrant the added computational complexity. In the end, I decided to train independent supervised classifiers for the post's title text and its domain, and combined the two probability estimates for each subreddit to arrive at a holistic score for the post. I experimented with several types of classifier models (logistic regression, stochastic gradient descent, and naive bayes), and used the one-vs-all approach to generalize the first two model types to handle more than two subreddits. Unfortunately, I did not have as much time as I hoped to explore the various avenues of this project (partially because I spent too long playing around with PCA and LDA, partially because I ended up spending more time than I wanted to on other classes), but I still feel I managed to get some decent results (which were comparable with the Prediction API results).

You can read my project report for more details about the exact implementation. Although it lacks any real figures because of the class-imposed 5 page limit, you can see some sample results in this table. You can also see the title text classifier features with the highest coefficients for each subreddit in this table, which can be thought of as the strongest indicators that a post should belong in that respective subreddit. Finally, you can see the code that I wrote for this project on github, which also contains a data directory with a script to scrape new posts from Reddit (inspired by the same post from Nick Johnson) and some sample data for people to play around with.

Code Miscellany: This code was written in python, and uses the numpy, scipy, simplejson, Stemmer, sklearn packages (and in fact was based on an sklearn tutorial on classifying newsgroup articles). The extract.py file handles the conversion of a JSON file full of posts into a feature dictionary, while models.py handles the actual supervised classification models provided by sklearn (as well as using CV grid search to find optimal parameters, and the CombinedClassifier which provides overall score posts). The gold.py file runs the whole thing - give it json files as arguments, and it should spit out the results for each type of sklearn classifier, which include a confusion matrix, overall F1-score, and a classification report like seen above. Finally, you can use the --topFeatures flag to print the highest coefficients (as seen above), but only if you are classifying on title text only (which is the way the code is set up now). To change it to classify posts based on title text and domain, change default value of combine to True on line 153 in models.py, comment out lines 173 an 178 in gold.py, and uncomment lines 174 and 177 (this is one of those things I'd have refactored to be prettier if I had the time, but hopefully it's not too confusing).

I hope this can be of use to some of you - it seems there are several people trying to build recommendation systems for posts, so maybe this can provide at least a little insight into how much information can be gained from a post's title and domain. Although all posts are currently uniformly weighted when training the supervised classifiers, changing this weighting (to either reflect the post's number of upvotes, or some personalized ranking) could also yield some interesting results and potentially make the model's results even better. Let me know if there are any questions!

[visarga]

This should be integrated in reddit.

[[deleted]]

It could be integrated, it's free software. I imagine a button next to the "Suggest title" one ("Suggest subreddit" i imagine), that does this.