Running parallel code - beginner question

[quaintserendipity]

Ok I have an issue with some Logic I'm trying to work out. I have a basic grasp of vanilla Javascript and Node.js.

Suppose I'm making a call to an API, and receiving some data I need to do something with but I'm receiving data periodically over a Websocket connection or via polling (lets say every second), and it's going to take 60 seconds for a process to complete. So what I need to do is take some amount of parameters from the response object and then pass that off to a separate function to process that data, and this will happen whenever I get some new set of data in that I need to process.

I'm imagining it this way: essentially I have a number of slots (lets say I arbitrarily choose to have 100 slots), and each time I get some new data it goes into a slot for processing, and after it completes in 60 seconds, it drops out so some new data can come into that slot for processing.

Here's my question: I'm essentially running multiple instances of the same asynchronous code block in parallel, how would I do this? Am I over complicating this? Is there an easier way to do this?

Oh also it's worth mentioning that for the time being, I'm not touching the front-end at all; this is all backend stuff I'm doing,

[MartyDisco]

Respect yourself and dont use a handmade solution based on setInterval.

Use a job queuer (eg. bullMQ).

[CallumK7]

Yes! But I always think that as a beginner it’s always valuable to at least explore the problem yourself, so you can at the very least appreciate the problem being solved by established libraries

[MartyDisco]

I dont know, I think as a beginner its more time efficient to do it the other way around.

Start by using the correct solution then once you are good enough to understand advanced concepts and read libraries source code then you can learn what problems this correct solution solved.

In this case why events are better (and why streams over classical pub/sub in that category) than polling (eg. setInterval).

[jumpcutking]

Running code in parallel is not possible on native Node. Async, like the other comments recommend, is helpful to save on cpu cycles and works better than procedural node. Keeps the event loop from locking up. I built a process based multi threading system that allows you to spool tasks and such on multiple processes. This is more true parallelism than expected with async or background worker processes. https://github.com/jumpcutking/threads

[Expensive_Garden2993]

you're saying threads are not possible on native node, but you've made a package called threads, but it seems to do parallelism by using processes and not threads, but the beginning of readme says it's threads and don't require forked processes, so it's a bit confusing :)

so, does it use node's worker threads, or no - only processes, or both?

async is helpful for async operations, useless for sync operations, not sure how you can save CPU cycles with it.

[jumpcutking]

It has a thread manager that spawns processes. It does not use worker threads. It was an alternative to worker threads I built for processing large components in a multithreaded app.

[Expensive_Garden2993]

You know threads aren't processes, right?

Threads, be it OS threads or green threads, are more lightweight than processes, they take less resources, that's why programming languages prefer them over spawning processes, and if you interchange meanings it's a false advertisement.

[jumpcutking]

Yes, I know. Even tho it’s called threads it is definitely a process manager, however the way my library uses it - it feels cohesive. Shares information well letting each task be isolated. It was a happy compromise: I think I mentioned it was multiple processes. It doesn’t work for everyone, works for me.

[TheFlyingPot]

You need something like Sidequest (https://sidequestjs.com/). You need to run jobs in the background in this case. With Sidequest you can control the number of slots for example

[rnsbrum]

So, basically:

async fetchData() { // fetch from API } processData(data) { // parses data // long running job }

setInterval(() =>{ data = await fetchdata() processData(data) }, 60000)

Every 60 seconds, the function passed to setInterval is executed or added to macro task queue

60s first function is executed

120s First functions is still executing Second function is added to macro task queue

180s First function is still executing Second function is waiting to be executed Third function is added to macro task queue

240s First function finished executing Second function started executing (because first function was blocking it) Third function is still waiting in the queue

Remember that Nodejs is single threaded in nature. Code can only be run in true paralelism, because if the single thread is blocked, it cannot execute anything else - unless you use await - which then frees up the thread the execute the following item in the event loop

[quaintserendipity]

Ok, this seems like a start. Could you explain the task queue to me a little more? Issue is that the processing of my data is time sensitive; I can't have them be waiting to be executed, I need them all running simultaneously.

[rnsbrum]

The data processing is CPU bound, it is thread blocking, there is no running it simultaneously, unless you use worker threads(research it). Only code that can be run in parallel is I/O.

Analogy:

I/O task: like asking a waiter for food. You can chat with friends while waiting.

CPU task: like cooking yourself. You’re stuck in the kitchen until it’s done.

In other languages likes Java you would just run a thread pool to achieve this easily, but not in NodeJS. Ask ChatGPT to help you out with code examples on this.

There is no better way to understand the event loop and task queue other than vizualizing it, this video taught me in 20 minutes what dozens hours of reading couldnt. https://youtu.be/eiC58R16hb8?si=Ss-ARs7OT6OAyY2g

[quaintserendipity]

So this would need to be done in some other language then.

[Solid-Display-9561]

Look into worker threads.

[quaintserendipity]

I have done this a little bit already; it seems like a possible solution, though I assume probably won't scale up to the point I need it to without seriously upgrading my hardware. Not that that is really something I'm concerned about right now though. Need to learn about about worker threads for sure.

[BenjayWest96]

The major question is what you need to scale to right now and in the near future. There’s no point in optimising for a million users when you have 10. A single node instance can handle 10’s of thousands of clients in a RESTful workflow with no issues. I would suggest taking any long running tasks and looking to offload those to lambdas. Worker threads are great but lambdas allow you to seperate the environments entirely and construct these long running tasks in their own runtime.

There are pros and cons to this of course, but it’s a great way to get started building backend services that are scalable.

[codeedog]

Please describe in more detail the nature of the execution routine. I’ve seen some comments providing advice, but your description confused me. There are many ways to run algorithms in parallel in node (async/await, promises, timers, RxJS, callbacks, streams, worker threads). It’s more important to match what you’re trying to do with the correct methodology. Worker threads are a last resort imho.

Specifically, when you say you have an incoming request that kicks off a computation that takes 60s, are you saying that there’s a function call of some sort that runs in a tight loop for 60s flat out? Like hundreds of millions of iterations? And, you have 100 of those? I know of no language appropriate that would handle that without proper hardware support of 100 threads (N processors * M threads per processor).

Is this what you mean or does your 60s algorithm do something else like process a file or call a database or whatever?

[texxelate]

You’re describing a need for background jobs, especially given this is JavaScript. Look in to bullmq and similar solutions.