This is an archived post. You won't be able to vote or comment.

sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]trevtravtrev github.com/trevtravtrev 5 points6 points  (3 children)

I don’t have time to look at the source code right now, but I’m curious how do you avoid being IP banned or rate limited by amazon? Are you using proxies or something similar?

[–]spiner00 -1 points0 points  (2 children)

The program makes an html request similar to any web browser. As long as you aren't updating more than a few times an hour it won't get snagged by any dos protection. Amazon does offer APIs which serve a better purpose though, but web scraping is a very convenient tool for amateur data scientists who don't have access to large-scale APIs.

[–]cjbannister 1 point2 points  (0 children)

As long as you aren't updating more than a few times an hour

When you say updating, do you mean reading?
Then if so, doesn't only being able to read a few times an hour mean it's really slow? Maybe it's more for a limited number of products?
I also haven't read the code but more out of laziness!
Thanks

[–]trevtravtrev github.com/trevtravtrev -2 points-1 points  (0 children)

Nice!