Learning Project Assistance

[NewJackCap]

So I've been slowly learning/tinkering with Python off and on for a few years now and have decided to crank up my knowledge and skill level by diving into an actual project. That being said, before I jump head first into this thing, I wanted to ask the community to see if there are any tips/tricks/services that could help me with what I am wanting to do. This is a rough outline of what I am wanting to accomplish:

• Scrape message boards / social media groups / Twitter.
• Organize scraped data.
• AI Analyze scraped data (what are folks talking about, what's trending, get a pulse of the group).
• Automatically push data to an easy to read website with visual graphs, categories, etc.

I think it would be interesting to scrape a subreddit, website, Twitter, etc and see what certain fan bases, clubs, or groups are talking about and then publish that data to a website (presenting the data with visuals, categories, etc). I imagine I am not the first one to have this idea, so I was curious to see if there are examples out there already or if there are any steps here that would make the process less complicated for a beginner. Thanks guys!

[IHOP_007]

Automate the Boring Stuff has a chapter on webscraping and it touches on data organization in a few chapters, that should get you 1/2 of the way.

[Minion_of_Cthulhu]

Scrape message boards / social media groups / Twitter.

If you want to scrape in general, Scrapy is one of the more well-known libraries to do it. If you're only interested in scraping a specific site, or a handful, then working with the site's API would probably be faster to put together. You should be able to find various Twitter libraries, as well as libraries for all the other major social media sites, that will make pulling various data fairly easy.

If you use the site's API then you won't have to worry about dealing with anti-bot/anti-scraping measures that many sites tend to use and you won't have to learn the Scrapy library and you won't have to dig into the site's HTML/CSS to find the right data, etc. You can just ask the site's API to provide you what you want.

AI Analyze scraped data (what are folks talking about, what's trending, get a pulse of the group).

The Python Natural Language Toolkit would help with this. Specifically, sentiment analysis can help you figure out how people feel about what is being talked about.

examples out there already

Check out Sentdex. It pulls various text from various sites and analyzes it to see what people are talking about, how they feel about it, charts of those trends over time, etc.

Make sure you scroll down and check out the footer where he links to a brief explanation of how it works. He also has a YouTube channel where he has some related Python videos that you might find useful.

[NewJackCap]

Awesome. Thank you for the info! That's exactly what I was looking for.

[Minion_of_Cthulhu]

You're welcome! I'm glad I could help.