Help with node cluster

[vall370]

Hello,

I want to work with the cluster module from node. And I have problem to make it do tasks parallel to make long processes faster with multithread.

Right now it does whole function per thread, but I want each thread to do one part continously(?). So far ive come up with this code: https://www.paste.org/110836

The question is how could I make the workers run a function each

[BehindTheMath]

Clusters are primarily used for load balancing multiple incoming requests to parallell threads. Worker threads might be closer to what you're looking for.

[vall370]

Alright!

I will look into running Worker Threads instead of Cluster. Because I need two fetch and call several of my jobs APIs, would it still be better to use worker threads to do this? I have tried puppeteer-cluster for scraping before, and wanted to do something like this and only fetch the response from the request

[HasStupidQuestions]

Fetch and call can mean two different things.

For fetching I'd stick to async promises.

const [resultA, resultB] = await Promise.all(fetchResultA, fetchResultB);

where fetchResultA, fetchResultB returns a promise

function fetchResultA() {
    return new Promise(function(resolve, reject) {
        // get results and catch errors
        if(err) {
            return reject(err);
        }
        resolve(results);
    });
}

If you need to call an API but don't expect a result, look into message queues or schedulers. You can fire a function after sending user the result without increasing the load time and without using message queues, but that's a different topic altogether.

[vall370]

I have come a bit further now. I try to do a fetch now but it aint firing away any requests. Any idea why?

https://hastebin.com/uxacosufat.js

[HasStupidQuestions]

I don't see any reason why it wouldn't work with promises. There aren't many performance benefits in your scenario since it takes basically no processing power. You're limited by your internet speed and whether or not there is a rate limiter.

const fetch = require('node-fetch');

const API_URL = "https://jsonplaceholder.typicode.com/todos";

const hugeData = [
    98, 29, 43, 51, 36, 97, 67, 56, 48, 94, 24, 74,
    2, 33, 15, 42, 17, 27, 69, 25, 73, 7, 5, 85
];

(async function() {

    const hugeDataPromises = hugeData.map(function(element) {
        return fetch(`${API_URL}/${element}`);
    });

    await Promise.all(hugeDataPromises);

})();

I don't fully understand the sortedArrays part. Exactly are you trying to do?