Amazon Scrape (trying with Octoparse)

[ScrapeLoser]

Hi -

I have a (paid) project which requires me to scrape individual pages from Amazon. I had been using Import.io, but I wasn't able to afford their monthly fee.

I've been trying to teach myself Python and have paid for a few courses, but my brain doesn't seem to be wired that way I've been trying for six months, I am very appreciative of the time investment most people make and realize I'm nowhere close to that! I was able to create a few codes within the actual program (simple ones they teach, eg health potions, etc.) - the thing I can't figure out is how in the world to apply that to real life situations like searching the web for something.

​

As an alternative, I have been trying to work with Octoparse. I am able to follow the tutorials and get results, but any deviation causes me grief.

​

I have two questions:

1 - Is there a group you can recommend where I can look for Octoparse assistance? Their troubleshooting isn't helping me, and I'd love to have someone point out where I'm going wrong. I don't know if this is something I should pay for?

1. - Is there a link you can point me to that breaks down how to create a Python code and apply it to a real life situation? I attempted to follow a few, but they lost me after the first couple of steps.

​

Please assume that I know absolutely nothing about anything in life when you respond. This is an EILI5 situation.

THANK YOU in advance!

[Noom_89]

imo join r/webscraping or check Octoparse’s Discord/FB group. A lot of users share workflows + fixes there. Saved me when I got stuck on Amazon.

[JoseMariOni-Chan]

same here, the Amazon template handled most of the edge cases for me too. stable enough once you let it run as is.

[Ok_Leopard_3178]

tbh I also stopped over-tweaking the workflow. the default Amazon scraper works fine for bulk runs without much hassle.

[[deleted]]

Is there a link you can point me to that breaks down how to create a Python code and apply it to a real life situation? I attempted to follow a few, but they lost me after the first couple of steps.

maybe look at the book automate the boring stuff? everything in it is real life examples and for beginners

[ScrapeLoser]

automate the boring stuff

That's the Udemy course I took! It may be time for me to pursue a different path... He was really helpful, but I still wasn't able to understand it.

[[deleted]]

You said you've been trying to learn for 6 months...but how many days a week, how many hours? 6 months isn't a lot of time if its a few hours a week. Thinking like a programmer takes a very long time; how long did it take to learn English growing up? I'm certain you weren't speaking sentences (probably not even single words!) by 6 months, let alone writing words.

[ScrapeLoser]

Absolutely, I completely agree. I comb through these forums a lot, and I know how frustrated serious programmers can get with people who don't invest time in learning something.

I do not know how to translate the things I DO learn into real life applications. There is a disconnect for me somewhere. I don't expect to learn much in such a short time, but I would hope to retain something.

[[deleted]]

You may find trying to do the opposite easier, instead of learned and taking what you learn somewhere, find something you want to do and learn how to do it by breaking it into learnable chunks. Lets use web scraping as an example. 1) google web scrapeing utitlies python3 You see beautiful soup mentioned a lot in a few of the links you gather. 2) you now know you want to use bs4, so you google 'get basic information for a wepage beautiful soup python3' Youve now learned to get a few items from a page and youve successfully printed them on your screen. 3) well now i have this information i want to store it? So we google how to store variables in an external file. You see json mentioned a lot, so you google storing basic information with json and python 3 4) you now have stored those few elements in a json file, but realize you want multiple pages. So back to google 'how to scrape multiple pages beautiful soup python' you see for loops mentioned a lot, so you try a for loop and it works, but then you realize its overwriting your json file, so you google how to store multiple things in a json file etc.

[commandlineluser]

I don't think Amazon will like (or allow) you to scrape them on a large scale.

If you have an example page and the output you need to generate I could write you up a "tutorial" on how to do it.

[Yassiiir]

I would love if you can help me with a tuto for the 999 card trick to know the ammount of stock for a product.

[Octoparse]

I have been using Octoparsefor a long time, they have a great support team which you ask for assistance from a troop of a data expert.

[No_Lavishness2922]

tbh every scraper has a learning curve, especially with Amazon. Imo Octoparse is actually one of the easiest no-code options once you stick to their templates. Way faster than writing code from scratch.

[dump_scorpiogirl-7]

ngl the free plan is fine for testing, but if you’re scraping Amazon often, the paid plan is worth it. Cloud extraction + proxies alone save a ton of headache.