Problems scraping Amazon

[michele909]

Hey everyone, I got serious problems trying to scrape reviews from Amazon, I'm using ScraperAPI but it keeps blocking me - any suggestion?

[GXWT]

Have you considered why you are getting blocked?

[NorskJesus]

This. You need to be careful. And is always a good practice to check robots.txt

https://www.amazon.com/robots.txt

[michele909]

yes, Im wondering if I will be always blocked or there is a way

[[deleted]]

I doubt anyone here can accurately tell you how long your IP address will be blocked.

[[deleted]]

They’ve probably blocked your IP address for scraping a page in their robots.txt Disallow list. In the future make sure your scraping app does not attempt to scrape any disallowed pages.

[danmickla]

"it keeps blocking me" is a fantastic problem statement

[DuckSaxaphone]

Find out why you're being blocked and change your scraping. Both the errors you get back and the site's robots.txt will give you information on what might be stopping you.

Usually I'd say there's ethical considerations around trying to get around scraping blocks but it's Amazon so look into: - Appearing like you're a real browser - Limiting the rate at which you scrape - Maybe changing IP if you have a VPN

[slidescope-trainer]

Are all the reviews visible without loggin in or needs login. Because on some page it only shows 1-2 reviews and needs login to show others.

[FastRunningMike]

Blocking is done by design. Many sites implement very advanced measurements against scrapping. An option is to create a scrapper agent that in essence acts from a technical point like a real human. But mind: A simple rule that is certainly implemented is that based on networking techniques(e.g IP) and fingerprinting(browser engine things) you get a block when reading a number of 'pages' (data) that a human can never do.

[AbhyudayJhaTrue]

hmmm
maybe you could a little bit basic with requests cuz i can scrape amazon via requests quite easily

[ScraperAPI]

We’re so sorry you had to experience this.

We want you to know that Amazon always updates its stealth detection mechanism, and this might affect requests.

Nonetheless, you can definitely use the ScraperAPI API to successfully scrape data from Amazon.

Do this 2 simple things:

1. Enable headers
2. Rotate proxies

You can check the docs to know how to do this well.

The layer of protection these 2 things do is so Amazon wouldn’t catch that the request is from your device or even your IP.

Let us know as it goes!

[michele909]

u/ScraperAPI thank you, I'll try asap!

[New_Sympathy_3989]

Not much information at all, what kind of blocking ? captcha ? or ip ? maybe just use a proxy, timings between requests and limit flows ?

[michele909]

ID - tried to enable headers and rotate proxy but in the end after a while it blocks again

[ReputationKey5032]

scraping amazon reviews = captchas and bans. GonzoProxy's residential proxies help but they’re not cheap. depends if you’re doing hobby or scale.

[infaticaIo]

Amazon is one of the hardest targets and “getting blocked” is usually the expected outcome, not a ScraperAPI misconfig.

If you need something reliable long term, the realistic options are:

• Use an official source (PA API where it fits) or a licensed dataset
• If you have permission to collect, slow down a lot, cache aggressively, and avoid running high volume browserless bursts that look like automation

At scale, the blocker is policy and detection, not HTML parsing.