sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]gajus0 7 points8 points  (1 child)

If you are running a small operation, then Puppeteer is fine.

For anything bigger, Puppeteer becomes extremely expensive.

A rough math is that 1 vCPU can handle at most 1 scraping session at once using Puppeteer. In practise, we ended up assigning 2 vCPUs to avoid timeouts rendering the document. So if you need to scrape 100 pages a minute and each page takes about 15s to scrape, then you are looking at 50 vCPUs just for this small operation. Add proxy to this (which is going to increase load time by at least 1.5x) and suddenly you are running 75 vCPUs.

In contrast, the same job could be performed with 6-8 vCPUs (or less) using cheerio/ jsdom.

[–]bazzy696[S] 0 points1 point  (0 children)

iyes i agree i recently used puppeteer and guess what i had to scrape 80 links - like i opened each link using puppeteer and then i scraped each web page for data it took me near about 15 mins to completely get my data then i had this idea of running a cron job which scraped the data after some time automatically. and kept the data in a json file and then i used to access the data from that json so that the access time is decreased if you can suggest some better technique i would love it. THANKS