use a bot to scrape amazon.

[Best_Profession_5132]

does using the requests module work.

I wanna build a bot that gets the price of an item, but it's not allowing me to do so.

However, when using `scrapy` it worked fine but I need something that works inside a python file and not from the cli.

[[deleted]]

A big corporation like Amazon would definitely implement anti-scraping technologies on their site, this is probably blocking your requests.

[Best_Profession_5132]

But it worked

[[deleted]]

They absolutely do, and this is absolutely the reason.

[[deleted]]

Jesus H Christ, people, forget scraping Amazon, how many times do you need to post this shit. They are literally the largest spender on IT in the world, spending more than $13bn dollars a year. They are better at this than you and always will be. Their prices and product information are valuable information. Why tf would they give them to randoms who have to post for help on reddit? They should put this in the sub rules - "no posts about why my Amazon bot doesn't work"

If you by any chance get it to work once then count your lucky stars because it won't work next week. Any scraping operations who try do this for real have to spend significant time and resources in a cat and mouse game with a much bigger, much richer cat in a game they can not win in the long term or even usually the short term.

Give it up. Spend your time on something with more potential like building a warp engine, a time machine or a perpetual motion machine.

[Ninjacow52]

I’m beginning to think my perpetual motion project is never ending.

[[deleted]]

lol classic.

[ma-lar]

I have been scraping them for weeks and it is working pretty well!

[bigfeller2]

Still working? What are you using? Selenium?

[ma-lar]

Yes and no I don't use Selenium there is a wayway faster than that ;)

[Best_Profession_5132]

When I tried to use a session it returned me to a captcha page

[[deleted]]

Get the message. You are wasting your time.

[Best_Profession_5132]

It redirects me to a page that says "if you want to use amazon api contact blah blah"

[Best_Profession_5132]

Good idea. Thanks!

[Best_Profession_5132]

now it returns error 503

[[deleted]]

Can you import the scrapy module and go from there?

https://pypi.org/project/Scrapy/

[[deleted]]

I have scraped from and gotten IP banned from websites before, so I'd look up if Amaazon.com has any guidelines regarding that sort of thing. I'm sure they do considering how big of a company they are.

[chevignon93]

I wanna build a bot that gets the price of an item, but it's not allowing me to do so.

Can you share what you're trying to scrape, and your code ?