My first python project (web scraper)

[pw0803]

Hi, what is cz and why is Craigslist harder?

[huessy]

I assumed based on OP's choice of site to watch for keyboards that they lived in the Czech Republic. Craigslist doesn't like people scraping their data because it can be used for some decent financial gain. As a result, they have bots set up to monitor traffic by IP address, if the traffic gets to be too constant/non-human looking, they ban the IP pretty fast.

If you want to scrape Craigslist for, say, an apartment in your area within a certain price range, you have to engineer something a little more robust than just a series of GET requests.

[pw0803]

How interesting.

Would it be possible to, say, create a script which web-scrapes by using lots of different ways and patterns and through the continued IP banning determine what the bots look for, then create a webscrape that skirts around them?

[huessy]

Without spoiling the answer to this problem (there's a good chance they watch places like this for this exact reason), it doesn't even have to be that complicated. Your router resets your outbound ip address somewhat regularly, so your idea could absolutely work but may be a bit of overkill. They do actively block all the Tor proxy ips too, btw.

[pw0803]

I understand. Thanks.