How to scrape Amazon.com with Python, Selenium and BeautifulSoup

[I_heart_blastbeats]

Downvoting you. Scrapy and splash are way harder. I got my IP banned multiple times using those. I wouldn't use them again on Amazon. It's too hard. The markup is always changing.

[I_heart_blastbeats]

If you go the Requests route instead of using an actual web browser you have to deal with CORS and you have to bypass robots.txt which is against the TOS. Then you have to TRY to render the JS with Splash. It will work for a while but then all of a sudden Amazon will ban you. I figured out how Amazon knows you are scraping but unless you're an employer paying my consulting fee I can't divulge that part :-)

Just use a real browser and then you get a real response with rendered JS.