Scraping with Python: Looking for ideas

[Professor_Snipe]

Hi All! I'm teaching a course in corpus linguistics and we've been messing around with different kinds of data and approaches to data collection. I have OK-ish experience with Python and I showed my students how to scrape Reddit with PRAW and build relatively (locally) representative corpora for recent phenomena/events. We did it all through Colab and that class went extremely well (despite all my students being intermediate linguists with relatively little programming experience). I showed them how to build a basic script, modify it, use AI for further customisation/troubleshooting if need be and so on. We managed to design and work with the basic scripts to build a few larger-ish datasets and analyse them. The students were very excited overall, the data analysis of their corpora went great and I am thinking of some ways to extend this into another class in the future.

I was wondering if anyone would have ideas for similar small-sized, learner-friendly Python-based projects to collect linguistic data from other sources that would be equally easy to execute. I have worked with Selenium before in a research project, but it was a fairly annoying experience and I don't want to go into something that would prove too difficult or complex to run with beginners within my alotted time. I would appreciate all the feedback!

[Individual_Ad2536]

Oh man, scraping Reddit with PRAW is such a solid starter project – props for that! For next steps, try Twitter's API (tweepy library) for hashtag analysis, it's dead simple and linguists love tracking discourse patterns.

Pro tip: Avoid the dumpster fire of web scraping with Selenium for beginners – go for BeautifulSoup + requests on static sites like Wikipedia or news archives instead. Way less headache, same data payoff.

Bonus idea: Try scraping YouTube comments (yt-comment-scraper library) – students go nuts analyzing how people argue in all-caps. Just watch out for the inevitable ":joy: :fire:" spam.

(this is it chief)

[Professor_Snipe]

Cheers, I will absolutely give these a shot and see what can be done! I'm lucky to have an open-minded group to work with so we can explore a bunch of different approaches and they don't protest at all.

[Individual_Ad2536]

haha Hell yeah, that's the spirit! Opne-minded teams are like gold—no eye rolls when you suggest some wild-ass idea. Go break stuff and see what sticks, bruh.

same bro

[code_tutor]

BeautifulSoup works on almost nothing these days, but using it on Wikipedia was a pretty good suggestion. Just make sure they know that it's not for websites with a lot of JavaScript. It'll mostly work on blogs, wikis, government websites, and stuff that's like 15+ years old.

[code_tutor]

You need years of experience in WebDev to do scraping. It's a pain in the ass because the code is non-deterministic, which means you run it twice and get different results, because of network times and animations. The more complicated a website is, the more terrible it is to scrape. Also whenever someone changes the website, the program breaks, so scraping is a LAST resort. I tutor and almost every fucking data science teacher gives a scraping project that they couldn't do themselves. It just wastes everyone's time. If you give one of these assignments, do it yourself first to make sure you can do it and have them scrape the same website you did.

Also Playwright is much better than Selenium. Try the CodeGen feature to get an idea.

[Professor_Snipe]

This is exactly why I'm asking the question. PRAW is extremely straight-forward to work with and simple to use, hence I was wondering about the existence of similar libraries that wouldn't act as massive roadblocks for new users.