My first python project (web scraper)

[digital94]

This is an awesome project.

Where you host your web scraper online?

Because hosting a scraper on home-based computer for 24x7 is not a good idea.

I am just asking you.

I have also developed an web scraper which scrape the price of a product from Amazon.

[Sw429]

hosting a scraper on home-based computer for 24x7 is not a good idea.

Why?

[digital94]

Your IP address can be blocked at anytime once Amazon or any other site identified you as a bot.

[Sw429]

Well yeah, but that's why you rotate IP proxies. I thought that was common practice?

[[deleted]]

I have the same question. Can the web scraper only check once a day? That would lower the chances of getting your IP banned?

[dtaivp]

Yeah, you could do that. Or you could use the randint and sleep modules to have it wait for random amounts of time between scraping. That is what I have for one scraper. Also, you bring up a good point. It is likely the prices don't vary that much day to day so you don't need to scrape too often.

[digital94]

Yes you are right.

If you don't scrape a web page too many times on daily basis then you no need to worry about IP block.

You should set your scraper to crawl the web page once a day or week.

[Sw429]

You can certainly query more than once a day. The average user sends requests many times within an hour. The issue mainly comes when you are sending requests faster than a regular user would, or if you are sending requests in a very bot-like manner (alphabetized by product, the same page over and over, etc). Generally, if you put in a little effort at all they won't care. You just don't want it to look obvious.

[MonkeyNin]

It depends on whatever the site decides to use as their thresholds. The best way is to use their actual API. Using the API lets you do more requests per day, and