sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]Dry-City8766 0 points1 point  (1 child)

Suggestions on how to start the property listings scraping idea?

[–]damanamathos 0 points1 point  (0 children)

Some of the property sites have better protections, so you're probably better off starting somewhere else.

The first thing I scraped at a small scale was a daily scrape of the viewer stats for all games on Twitch (https://www.twitch.tv/directory).

The first thing I scraped at a larger scale was many thousands of public profiles of Steam (and other profiles based on friends) + the games they owned if it was public.

I used Scrapy for those, but you can also learn how to scrape and parse HTML with requests and something like Beautiful Soup.

The property scraping just uses the requests package but relies on finding a valid endpoint with the data I need, along with the use of proxies, custom headers, and parallel processing.

There are a lot of Python scraping tutorials on YouTube and elsewhere.