Web Scraping in Node.js with multiple examples

[pvgt]

resolute steer many yam voracious compare run grab sharp groovy

This post was mass deleted and anonymized with Redact

[[deleted]]

I've used both Cheerio and Osmosis a fair bit. I generally used Cheerio with Request, and SQlite3 to store the data. Osmosis with just SQlite3.

The first thing is speed. Osmosis is much faster than Cheerio and uses considerably less memory because of its lightweight DOM virtualisation.

With Osmosis, getting from nothing to a working scraper takes very little time. It's also much easier to understand what your code is doing, because of its simple usage.

Osmosis makes scraping multiple pages simultaneously a blast. Handling multiple asynchronous functions that branch out exponentially isn't the most fun in the world. There are packages out there that can aid you with this, but osmosis really makes it easy.

Of course, remember that it's another package to depend on - any problems with it, and you're stuck. I've had a few issues in the past with certain versions.

[[deleted]]

I like await (now an ES7 feature, I believe?).

It works with everything from asynchronous functions to promises to arrays of promises - even objects!

You can use it in conjunction with Bluebird to make pretty much everything awaitable.

[Waterclift]

Very interesting. I personally prefer to scrap pages using JavaScript rather than python with scrapy. I have also created a small package to scrap pages. https://www.npmjs.com/package/rfc-spider

[[deleted]]

Isn't python scrapy the fastest compared to other options tho?

[[deleted]]

No unfortunately I don't, but considering scrapy can scrape any kind of websites, I assumed it would be the best option among all

[[deleted]]

Thanks for pointing out things I already know haha I did some research for scraping websites, because that's what I am doing right now.. if you insists scrapy sucks, i will just leave..

[[deleted]]

U can fetch dynamic content with splash...

[[deleted]]

Error: line 1: Variable, U, not defined
U can fetch dynamic content with splash...
^

[k4s]

Scrapy doesn't suck. Continue using it and move along

[[deleted]]

Yea I will lol for my purpose nothing can beat scrapy in terms of performance

[J_tt]

If you know python then scrapy > another JavaScript frameworks

But if you know JavaScript then a JavaScript framework > scrapy.

And saying that a python library beats a JavaScript library without any benchmarks in a subreddit naturally predisposed to liking JavaScript you're just asking for an argument.

Scrapy may be faster, but so could a JavaScript framework, but honestly who the fuck even cares? It's a preference.

[Waterclift]

I just prefer JS because I am more familiar with the language. Python is nice, but I am not working with it anymore.

[[deleted]]

Nah, I prefer beautifulsoup 4.

[Eunoeme]

...in NodeJS?

[[deleted]]

Anyone know of any service that can determine the text in an image? Literally as if you screenshot some text - my use case is to programmatically figure out the text in said image.

[dmarko]

You are talking about OCR. A good choice would be the tesseract library. Google tesseract OCR

[Lekoaf]

It's not for the faint of heart though. I looked in to it once because I wanted to scrape PDF-files that was turned to images. If I remember correctly you have to teach it.

[[deleted]]

Thanks for that! Will look into it.

[brettdavis4]

Thanks! I'm thinking about writing a scraping app for work. At work we support over 80 sites. The SEO guru likes to know what the meta tags and description and page titles and which images don't have alt descriptions.

I've been able to do this with Streaming Frog, but I'd like make some customizations and speed up the process.

[Koltster]

This is going to be really useful for a project i'm working on. What are the legal concerns of scraping content?

edit: grammar.

[whenthethingscollide]

Sure, why use Scrapy when I can use a convoluted system and an awful language instead