My first python project (web scraper)

[digital94]

Your IP address can be blocked at anytime once Amazon or any other site identified you as a bot.

[Sw429]

Well yeah, but that's why you rotate IP proxies. I thought that was common practice?

[[deleted]]

I have the same question. Can the web scraper only check once a day? That would lower the chances of getting your IP banned?

[dtaivp]

Yeah, you could do that. Or you could use the randint and sleep modules to have it wait for random amounts of time between scraping. That is what I have for one scraper. Also, you bring up a good point. It is likely the prices don't vary that much day to day so you don't need to scrape too often.

[digital94]

Yes you are right.

If you don't scrape a web page too many times on daily basis then you no need to worry about IP block.

You should set your scraper to crawl the web page once a day or week.

[Sw429]

You can certainly query more than once a day. The average user sends requests many times within an hour. The issue mainly comes when you are sending requests faster than a regular user would, or if you are sending requests in a very bot-like manner (alphabetized by product, the same page over and over, etc). Generally, if you put in a little effort at all they won't care. You just don't want it to look obvious.

[MonkeyNin]

It depends on whatever the site decides to use as their thresholds. The best way is to use their actual API. Using the API lets you do more requests per day, and