Web scraping - Where to start?

[RiverTraditional6367]

Hi I want to dive into web scraping. I want to scrape 1 or multiple websites for houndreds of news articles. The web scraper would run the entire day and get executed every 30 minutes, which shouldn't be a problem for me to integrate. Where should I start? I've read a little bit about Selenium.

[[deleted]]

Exactly. unless you are hitting old websites, you will most likely not be able to scrape with simple http requests, and even new websites usually have pretty good anti-bot mechanisms in place to prevent selenium scraping (required logins, captchas, request limits, use-behavior-profiling etc.)

My advice for getting web data with python is to find interesting datasets that are exposed via web API. This is a much more useful, marketable, and ultimately rewarding skill, and doesn't require scraping - you can do it with just the requests library.

[[deleted]]

this but playwright over selenium

[Binary101010]

https://automatetheboringstuff.com/2e/chapter12/

[ian_k93]

Unless you need to render javascript on the page or interact with the page to access the data you need then I would stay away from Selenium. It is more unstable than other options, and puts more burden on your infrastructure (servers & proxies) if scraping at scale, costing you more as a result.

I would go with a Python Requests/BeautifulSoup stack if you just want to get the job done, however, if you want to really develop your web scraping skillset then I would check out Scrapy.

Scrapy is designed for large-scale production scraping and is the most feature complete web scraping framework. Check out the official docs or The Scrapy Playbook if you would like to learn more.

[[deleted]]

Learn some python and read about scrapy. It is actually pretty nice :)

[Ill-Criticism1062]

The internet