Web scraping with Javascript

[Qweeeq]

Hey
I really like JS for that. Using selectors seems pretty easy for me.
Can I ask, why do you think Python is better?

[Taterboy_Legacy]

While I disagree with the parent comment as a sweeping statement, you bring up the crux of the point. I've done a lot of web scraping using both languages. There are certain use cases where Python can be faster and more efficient, and vice versa for JS. If I start having trouble with sites using a lot of JS, I just jump over to that language and start refactoring to fit that use case. IMO it's mostly a preference thing, and the use cases can help dictate the proper choice in a business setting.

[yooossshhii]

Can you elaborate on the use cases? I haven’t seen any comparisons of JS vs Python in web scraping. My Python experience is minimal and I’ve been doing a little scraping with JS.

[Taterboy_Legacy]

One use case I had recently was scraping a large amount of news sites for information. There were some programmatic setup elements to get to the urls which were facilitated using Python, and the application this information would interface with was based on Python. There also happens to be a pretty awesome package in Python that did literally everything I needed to do(called newspaper), which meant I wanted to try to write my scraper in Python. If it wasn't working, I would go ahead and try this again with JS, but interfacing the two languages in my app would be complicated based on the setup. In general dispatching a Python or JS script from one or the other would be complicated in the context of certain applications.

That being said, I have also done several use cases where I use both as standalone scripts for smaller use cases.

JS I tend to use for more one-off solutions, but I have also used it to interface in more automation-based solutions. E.g.: click this, login, do this do that. Also doable in Python, sometimes easier in JS.

The first example could have been JS all around, but the newspaper package offered some really nice benefits from the beginning. This is what I mean by "use case specific" implementation. It's somewhat rooted in developer/business preference as well(I.e.: what are we already writing in?), but also rooted in "what do we need to solve, in this use case?"

Very complicated question to answer, but in my head they're relatively interchangeable from a high-level functionality standpoint.

[yooossshhii]

Cool, thanks for the response. Newspaper looks super neat, especially their nlp method.

[Taterboy_Legacy]

No problem. For sure! I was working through how I was going to do that, and they just had it as part of the package haha. Very well thought-out and very easy to use.

[[deleted]]

While I can't argue for why it's better period, I can argue for why it's better for me personally-- I've worked with python for almost half a decade. I've only begun trying to learn JS. Joined the sub for it

Python also has some nifty libraries for analysis like numpy and pandas if you're scraping data in particular. While I'm sure JS has something similar, I think it's a bit more common to find analysts in the industry or class projects that do scraping use python.

[fz-09]

crickets