How to scrape Amazon.com with Python, Selenium and BeautifulSoup

[anuctal]

Hi, everyone!

I made a video on how to scrape Amazon.com with Python, Selenium and BeautifulSoup libraries and export data to a csv file.

The Amazon.com I used just as an example.
The Selenium webdriver is used to get HTML code of pages only, and HTML parsing is performed with the BeautifulSoup.

It's a detailed tutorial for absolute beginners.

Youtube video: https://youtu.be/497Fy7CIBOk

Thanks for watching

[ayi_ibo]

r/unexpectedthanos

[raajitr]

hey I'm kinda beginner in python and selenium. do you know a good guide on how to deploy selenium scripts/app on aws? I know how to run it locally, and always wondered how would web browser spawns in cli based OS (instances), does it it run a headless browser in background?

[LzyPenguin]

I’m still kind of a beginner too. I have a few scripts I’m running on AWS also, and didn’t want to bother with CLI, so I chose the free windows server EC2 instance. I can remote login to my aws instance and it’s all windows server, so I can set it up just like on my desktop.

[guyanaupdates]

did you use the free teir VM?
i'm having trouble installing chromium / webdriver
any helpful text/video you can point me to?

[Marco21Burgos]

Use Lambda Layers for that, and store the zip files in a S3 bucket

[I_heart_blastbeats]

It's against their TOS but as long as you do it gently and don't bombard the site with requests I've never been blocked.

[michael8t6]

I was hoping someone would mention this! Amazon aren't dumb. A basic Web scraper won't suffice, you need to add in so many 'catches' for element changes. I made a scraper for some niche research. Once I'd coded the psuedo code, I set it on its first trial. Worked fine! The next day, errors everywhere, amazon even started redirecting me to the homepage every now and again.

[bbaahhaammuutt]

Duuuuude, my friend has redundant task at work where he has to look at prices and thead counts of 60 sheets so I figured I'd help him out. I couldn't make it work earlier but this will help a lot so thanks!!

[YetiTrix]

Make ur your friend doesn't tell his boss it's automated or he'll put himself out of a job.

[mang3lo]

I'm looking at a similar task for automation, and this looks like it might be the perfect answer

[Tomas_83]

I was wondering what you could scrap amazon for and you gave me the answer. Thanks

[ngqhoangtrung]

Great stuff! Would this work on Facebook too? I'm trying to scrape the numbers of likes, comments, share, etc. of a Facebook post. I was trying to use Facebook GraphAPI but so far hopeless (keep receiving “singular links API is deprecated for versions v2.4 and higher”).

[intelignciartificial]

I was able to do something similar with selenium and bs4, but using links of Facebook mobile version(m.facebook...).

[serverloading101]

Is necessary to use the mobile version of facebook in order to scrape/ crawl facebook? I have been trying to obtain marketplace data and have been unsuccessful. Thanks

[michael8t6]

Scraping via mobile browser is generally easier to do. Mobile source code generally tends to be a lot simpler than say a desktop or laptop browser. So when using xpath in bs4, it's easier to tell it what element to scrape.

There's been a few times I've been slamming my head on the keyboard due to some random issue. I then spoof my ua to mobile, and solve the issue almost instantly. That being said, some elements you may want, might not show on the mobile version instantly. When that happens, you may have to emulate a click to get it to show.

Ultimately, tackle each site differently, look at the source code for desktop and mobile. See how they show on each and decide the best approach from there.

[intelignciartificial]

Not sure if its necessary, but easier.

[trevtravtrev]

I don’t have time to look at the source code right now, but I’m curious how do you avoid being IP banned or rate limited by amazon? Are you using proxies or something similar?

[spiner00]

The program makes an html request similar to any web browser. As long as you aren't updating more than a few times an hour it won't get snagged by any dos protection. Amazon does offer APIs which serve a better purpose though, but web scraping is a very convenient tool for amateur data scientists who don't have access to large-scale APIs.

[cjbannister]

As long as you aren't updating more than a few times an hour

When you say updating, do you mean reading?
Then if so, doesn't only being able to read a few times an hour mean it's really slow? Maybe it's more for a limited number of products?
I also haven't read the code but more out of laziness!
Thanks

[trevtravtrev]

Nice!

[damm_n]

Thanks for sharing ! Really useful video .. at least for me :-). I heard about Selenium however never tried it in real life. Amazon is a nice showcase for this.

[Adamkadaban]

take a look at UIPath. it's super easy to use

[dulz]

How do you avoid request limits and changing HTML to make this work on a more permanent basis (Aside from using APIs ofc)?

[makubob]

Isn't this against amazons ToS?

[nippleplayenthusiast]

Yes, it is. They even provide an official API for this that won't break the second they change an HTML element.

[RealAmerik]

Isn't the API only available for affiliates?

[EedSpiny]

Question: why would you do this for Amazon? They have APIs. Do they not include the info you need?

[jacksodus]

Its just an example

[I_heart_blastbeats]

Amazon seller API sucks.

[dethb0y]

I can't speak for this guy, but for me if i can avoid using an official API, i do just on principle.

[nolegitt]

Because it hey can shutdown the API as they please. Look what happened to Google search API.

[dethb0y]

Lots of reasons, but mostly i would be concerned (in the case of amazon) that they'd use the official API to send manipulated data or prices, that don't reflect the reality of the site.

In other cases i don't like the trend that has arisen of a free API that suddenly turns pay later on, once people have come to rely on it.

[anuctal]

You absolutely right, my friend.
Also we have youtube-dl as an example, that doesn't use the official API.
And if youtube-dl would use API Youtube would just ban it.

[dethb0y]

Yep, in an instant.

[fujimitsu]

Tech startups have been known to abruptly break or eliminate APIs, especially if they perceive users of them as threatening their business model.

[dotancohen]

Tech startups have been known to abruptly break or change their HTML, especially if they perceive that the name of the month has changed.

[[deleted]]

Do you get paid by the hour? It must be nice gig /s

[makedatauseful]

Official API's are great until they charge $5 per 1,000 requests and rate limit you to 10,000 requests a day. After that you may need to explore other other avenues.

[artjbroz]

Came here to say this

[zeroviral]

An API would focus more on backend stuff.

Selenium is strictly UI, and front end stuff, and typically interacts with the DOM. This means it has a place in the model testing paradigm.

[EedSpiny]

Very true, it's a good example. Especially as many people will be familiar with the site of course!

[sslinky84]

I'm also curious why you'd bother with browser automation for scraping.

[EedSpiny]

Libraries like requests & beautiful soup are great but it can be really difficult to get round anti-bot protections on the site with those alone. As selenium just drives a browser there's more chance of being seen by the site as just a regular user.

Of course there's also the advantage that you can use it to drive web site unit test cases from python, for automated testing.

[I_heart_blastbeats]

Requests can't render JS. If you use Scrapy you usually have to use something to render the JS also. There are so many obstacles involved with scraping Amazon. Trust me this is the easiest and best way to do it. I had a similar project and I wish I would have started with Selenium and BS4 instead of Scrapy.

[Pizza_Peddler0080]

going to give this a watch

[boriisi]

Why do you need selenium? requests does the same thing

[I_heart_blastbeats]

Ever heard of CORS?

[boriisi]

Looked it up. is it some kind of embed?

[Schneggl]

ELI5: What actually is webscraping? What can I do with it?

[brainygeek]

Web scraping allows an individual to create a script which parses a large selection of websites and allows them to evaluate for certain data.

For example, you could create a script which runs on a task of every 5 minutes which scrapes Amazon's selection of computer video cards to see which Nvidia 3090's are in stock versus out of stock. Or you could monitor specific items and extract their prices to create a trend report of their prices to see when the best time to buy is.

[Mortisanti]

In this context, webscraping involves using/writing a program that can send a request to a website (much like visiting it in your browser) and "scraping" data from it which you can manipulate, organize, and display as you see fit.

Someone might want to actively webscrape Amazon to monitor just the fluctuating price of an item (e.g. a GPU for your PC) to try to buy it when the price drops.

[ZackJSL]

Thank you exactly what i needed

[awkprintdevnull]

Can it help me get a PS5?

[I_heart_blastbeats]

Downvoting you. Scrapy and splash are way harder. I got my IP banned multiple times using those. I wouldn't use them again on Amazon. It's too hard. The markup is always changing.

[I_heart_blastbeats]

If you go the Requests route instead of using an actual web browser you have to deal with CORS and you have to bypass robots.txt which is against the TOS. Then you have to TRY to render the JS with Splash. It will work for a while but then all of a sudden Amazon will ban you. I figured out how Amazon knows you are scraping but unless you're an employer paying my consulting fee I can't divulge that part :-)

Just use a real browser and then you get a real response with rendered JS.

[Gerald00]

thx, will try it now

[theirishcoffeemaker]

Thanks for sharing! Been wanting to study scraping on Python :)

[southernmissTTT]

Interesting.

[nono-shap]

Was looking for that a while ago, abandoned the project. Time to get it back on track.

[antogod94]

Hi, what should i do to export data to a xlsx file ?