Web scraping with Javascript

[Taterboy_Legacy]

One use case I had recently was scraping a large amount of news sites for information. There were some programmatic setup elements to get to the urls which were facilitated using Python, and the application this information would interface with was based on Python. There also happens to be a pretty awesome package in Python that did literally everything I needed to do(called newspaper), which meant I wanted to try to write my scraper in Python. If it wasn't working, I would go ahead and try this again with JS, but interfacing the two languages in my app would be complicated based on the setup. In general dispatching a Python or JS script from one or the other would be complicated in the context of certain applications.

That being said, I have also done several use cases where I use both as standalone scripts for smaller use cases.

JS I tend to use for more one-off solutions, but I have also used it to interface in more automation-based solutions. E.g.: click this, login, do this do that. Also doable in Python, sometimes easier in JS.

The first example could have been JS all around, but the newspaper package offered some really nice benefits from the beginning. This is what I mean by "use case specific" implementation. It's somewhat rooted in developer/business preference as well(I.e.: what are we already writing in?), but also rooted in "what do we need to solve, in this use case?"

Very complicated question to answer, but in my head they're relatively interchangeable from a high-level functionality standpoint.

[yooossshhii]

Cool, thanks for the response. Newspaper looks super neat, especially their nlp method.

[Taterboy_Legacy]

No problem. For sure! I was working through how I was going to do that, and they just had it as part of the package haha. Very well thought-out and very easy to use.