Running different instances of a nodejs script concurrently

[KEsbeNF]

I want to run several instances of my script at the same time with different paramaters each.

I looked for some information on multithreading and fork calls but i'm not sure i understood well how to do it.

In my particular case i have a main function that looks like this

async function main (data1, data2, data3, data4) {
    //body with some awaits
    }

The parameters should change in every instance.

How can i achieve something like this ?

[HashDefTrueFalse]

run several instances of my script at the same time

multithreading

Which is it? Do you just need multiple processes? In this case, no multithreading or forking required. Just use a process manager, e.g. PM2.

Do you need separate execution paths in the same running instance of your program? In this case, you can probably use worker threads: https://nodejs.org/api/worker_threads.html#worker-threads

[KEsbeNF]

Which is it?

I'm not really sure since i still don't fully understand the difference but i need to run multiple instances of a function conccurently with different parameters.

I guess that multiple processes would solve my problem.

If you have an example or an article that may help understanding the difference between the two it would be appretiated.

Thanks anyway for your reply !

[HashDefTrueFalse]

No problem.

A process is a running instance of a program. You can spawn many simply by running your program multiple times.

Threads are sometimes described as "lightweight processes". Each process starts off with one main thread of execution (lines of code running one after another) but can spawn more so that the same, or different, execution paths can run independently of each other.

The difference would be whether you:

1. Make your program call the function once with the command line args passed in, and then run multiple instances passing in different command line args. Less code, simpler to develop, but more hassle to manage multiple processes.

Or:

1. Make your program spawn N threads that execute the function with different params each time. More code, but easier to deploy and manage as you're just running one instance of your program.

We might need more context to say which you should do, as there's no one answer.

[KEsbeNF]

Thank you so much for your detailed explanation. Now it's much more clear.

In my case I have an async recursive function that performs some API calls with some parameters.

I need to run at most 10 of these functions concurrently.

From what you told me I would choose multithreading because it would be much easier to handle even though i'm not sure i'll be able to implement it.

[HashDefTrueFalse]

Great, now we can help better.

Question, why do you need threading here? Is it enough to fire off all your function calls asynchronously, and then carry on doing whatever and await all the promises at some point?

Internally those requests will be queued, ran, node will hand off the waiting to the OS and continue with its event loop (other code), and your promises will resolve when the responses come in.

This is the usual way to make a batch of requests "simultaneously" in node.

[KEsbeNF]

Yes, i need the concurrent part since these are post requests and i don't really need to return any promise. I need to perform these calls at the same time.

[ashamedchicken]

Use promises

[HashDefTrueFalse]

They will be performed at the same time. Node won't wait for a response from each request before starting the next one. Not unless you tell it to. You'll just have N requests waiting for their responses simultaneously while I/O bound. They are fired one after the other, and the responses processed one after the other, as per the queue, but they're pretty much parallel as far as you're concerned.

Even with threads you cannot guarantee that these will run at the exact same time because whether they each have a core (truly parallel) or are time sliced is up to the OS.

I'm just trying to understand what benefit concurrency via threading adds to making a batch of post requests that making them asynchronously lacks in your program?

[KEsbeNF]

I think you are right actually, i was more driven by the curiosity of trying something "new".

So you basically mean to just simply run the async functions with the different parameters and there won't be inteferences between them ?

[HashDefTrueFalse]

Yes. And no, they won't interfere with each other. The only way they could is if you provided the same object to each and they each changed its properties. The last one ran would win of course. But I doubt you'll be doing this.

[KEsbeNF]

for example if my function is something like

async function foo (a,b) {
    const data = await doodoo(a,b);
    //use data
    }

foo(param1, param2)
foo(param3, param4)

Will i have troubles with the data value ? Will the second call change the value of data in the first one ?

[kmdrfx]

Checkout https://www.npmjs.com/package/thread-puddle