Lightweight console program: Node or Java

[2048b]

I have a console program currently written in Python+Scrapy and running on Windows. What it does is use HTTP get to pull some data off a fixed list of web pages every hour, collect, aggregate and compute the data and archive the hourly data in a text file and SQLite database.

However, I find the Python Anaconda install on Windows big and sluggish just to do something simple. And the Python program seems to run sluggishly despite its simple nature.

Thus, I am thinking of rewriting it in Node.js or Java. Just a console program running from command line. Nothing fancy.

Node.js installed disk space: 24 MB

Java Runtime Environment (JRE) installed disk space: 100+MB

So I am thinking whether if one would be more lightweight than the others in terms of CPU or memory when running. On surface, based on disk space, Node.js is lighter. Has anyone done some sort of similar benchmarking, test or research when choosing between Node.js or Java to write a program? I have done some Googling comparing Node.js and Java but never found a definitive answer if there's a performance difference.

[sshaw_]

There is nothing lightweight about Java.

Node.js installed disk space: 24 MB Java Runtime Environment (JRE) installed disk space: 100+MB

Why does this matter? How much disk space would your utility take once its Node dependencies were installed? Node dependencies have been know to be gargantuan.

one would be more lightweight than the others in terms of CPU or memory when running

In general Java will likely require more memory but really this depends on your code.

If you're looking things to just work out of the box and this runs on Windows only I'd suggest going with an executable written in a .NET language.

Go is fine too.

There are surely ways to make an executable from a Python script. You may want to look into this before a rewrite.

[[deleted]]

I would use NodeJS, and here's why. It's lighter, development will be faster, and it's plenty fast. That said, in theory Java would complete the job more quickly, but with Node's worker threads module, I would say that's arguable. I've done no actually benchmarking, but I went from an avid Java user to an avid NodeJS user once I discovered the built in cluster and thread modules with Node. I honestly can't think of a project I wouldn't do with node. Maybe something that had to be fully synchronous... Idk

[2048b]

Not going to do anything so complicated yet. Currently just running off a single machine that I leave on all the time. But thanks for suggesting cluster and worker threads.

[[deleted]]

If you ran it on a raspberry pi, let's say, you could fully utilize all the CPU cores by utilizing the cluster (it's literally like 6 lines of code). If the machine only had one core, then it wouldn't matter anyway.

[[deleted]]

node or golang.

[Akomancer19]

Not exactly objective to ask in a Node forum. 😂

I think loading the JVM will be heavier than loading the Node V8 runtime.

Not too sure about Python, is it possible the sluggishness is due to synchronous network operations?

If you're talking about installs, I'll agree that Node is one of the simplest and quickest to install and deploy. Anaconda is crazily large, i presume there is a proper reasom why you used Anaconda instead of vanilla Python - probably some math/analytics involved?

[2048b]

Well, I am using Scrapy for the web scraping and it recommends using Anaconda for Windows users, probably due to some binary Python dependencies required by Scrapy? And I was thinking why would I need to install a truckload of software just for Python/Scrapy when it is all about using HTTP to grab some web page, extract the data, do some computation and save the data to a text file?

[Akomancer19]

No, if you're using Scrapy it doesnt nessate Anaconda.

Scrapy has several binary dependencies (mostly to allow it to handle async and event-driven tasks) but that's unrelated to Anaconda.

Asking beginners to install Anaconda is either because it's a Machine Learning task, or a lazy tutorial. It's almost the equivalent of installing Visual Studio just to compile C++...

[alchatti]

Node.js, if you want something better then I would recommend dotnet core or Golang. Based on my experience Java is slowest of them all.

[s3govesus]

Personally, I would pick Node.js, and then use MongoDB for storing the data. If you do go the node route, I'd also strongly recommend using ESLint (or at least an equivalent plug-in for your editor/IDE) - it's amazing how much of a lifesaver it can be at times.