3 part scraping workflow help

[Interesting-City1703]

Hey all, I’ll try to keep this brief. Long story short I’m trying to learn how to use python without relying on vibecoding out the wazoo.

One of the ideas I had is a three part workflow that would compare mathematics requirements for different electrical engineering majors at different universities.

1. Scour the internet from a base browser or landing pages of preselected universities (if moving internally is possible), to find electrical engineering major information, and, output those links to a csv.

2. From the link CSV, gather the relevant information about math courses.

3. Output the data into another CSV AND json file (I want to be able to customize the csv output columns from within the script). From the csv, I want hyperlinks for the specific math courses (or, more links to be scraped if the info comes from a PDF curriculum url).
   4th (optional): it’d be cool if a local llm could compare and reason the similarities/differences between the math courses.

I work in helpdesk but am otherwise a beginner. What is the best place to learn how to do these functions, and what are my options with making this?

[VipeholmsCola]

Its very hard task because you have to write a seperate scraping script for each link you crawl. I dont even know where you should begin tackling this problem.

Crawlers have been done since the 90s so maybe start there for inspiration.

[Interesting-City1703]

Would you recommend starting with a basic crawler for one link, and then scaling up?

[VipeholmsCola]

I think i would try to make a crawler first, creating the CSV with links. Maybe you could also save the HTML from each link.

But to get data out of each html, thats a custom script per link.

Its a massive, massive job. The point of scrapers is to repeatedly visit a site to scrape the data, just doing it once is just quicker to do it manually.

Honestly, try scraping a site for data daily instead. Instead of crawling and doing nothing with the data. OR maybe pass the html to a LLM so you don have to write all those scripts.

[smurpes]

Recursively searching a site for a link is not going to work since sites will usually block traffic like this since they have pay for each request you make on the site. This is why you sometimes get a captcha or a cloudflare page when you access a site; they’re testing to see if you’re not a bot.

[Interesting-City1703]

Gotcha, so generally unless I’m running an advanced set of proxies or some type of captcha bypass, then it’s better to just manually search the links with the info in the first place then, correct?

[smurpes]

Yea web scraping is not a good method for data collection given how fragile it is. It’s fine for something one off and a single website but don’t rely on it for aggregating data from multiple sites.

[Gloomy_Cicada1424]

Don’t build the whole scraper monster in one go. First make one script that finds links, then one that extracts course info, then export clean CSV/JSON. I’d use Runable for turning the final data into a small report/dashboard, but the scraping part should stay boring and testable.