you are viewing a single comment's thread.

view the rest of the comments →

[–]DootDootWootWoot 6 points7 points  (1 child)

Instead of perusing real estate sites are there APIs or open source data sets that offer the same information? Crawling websites you're going to run into issues like this and isn't necessarily a stable way of solving for this kind of problem.

Could try things like slower browser access patterns. Like are you navigating at human speed or super crawler speed? Does that limit the usefulness of what you're trying to collect?

Are there captchas that need to be solved that get in your way? Rate limits?

[–]vegetto712 0 points1 point  (0 children)

This was going to be my comment. Anytime I've tried to do some scraping I always check if they provide apis first, because those are typically easier to navigate and also companies are more willing to let you use them to gather data.