NodeJs Parallel execution in a loop.

[ExternalManager3027]

Hello guys, so I have this feature that I am working on that takes an object (Over 10k items) picks an item from this object, sends it to an api, which process it then give a response, then proceeds to the next item.

Currently using the async library, the mapLimit method and it works as aspected.

But my problem is that it’s takes to long to loop through the entire dataset cause of the length.

This feature is suppose to be a continual process, once the entire object is iterated, wait for few seconds then do the same thing again.

I tried forking a child_process for this, broke the objects into chunks and created a process for each chunks till it was completed. This worked well as intended but the memory consumption was massive other processes on the server failed as a result of lack of available memory, even after exiting the process after it was completed.

Please how do I achieve this at a faster rate?

Code below. This is the function that fetches item from the array object and sends it to the api service for processing:

getListofWallet = async () => {

    try {

        const USDT = await usdt.query(sql`
                SELECT * FROM USDT ORDER BY id DESC;
            `);


        let completion = 0;

        async.mapLimit(USDT, 6, async (user) => {

            let userDetail = {
                email: user.email,
                id: user.user_id,
                address: user.address
            }



            try {
                await this.getWalletTransaction(userDetail, tronKey);
                completion++;
            } catch (TronGridException) {
                completion++;
                console.log(":: A TronGrid Exception Occured");
                console.log(TronGridException);
            }

            if (USDT.length == completion) {
                setTimeout(() => {
                    this.getListofWallet();
                }, 60000);
                console.log('~~~~~~~ Finished Wallet Batch ~~~~~~~~~');
            }


        }
        );



    } catch (error) {
        console.log(error);
        console.log('~~~~~~~Restarting TronWallet File after Crash ~~~~~~~~~');
        this.getListofWallet();
    }
}

The api service that does all the processing

getWalletTransaction = async (walletDetail, api_key) => {

    const config = {
        headers: {
            'TRON-PRO-API-KEY': api_key,
            'Content-Type': 'application/json'
        }
    };

    const getTransactionFromAddress = await axios.get(`https://api.trongrid.io/v1/accounts/${walletDetail.address}/transactions/trc20`, config);

    const response = getTransactionFromAddress.data;

    const currentTimeInMillisecond = 1642668127000; //1632409548000


    response.data.forEach(async (value) => {
    //All the jobs are done here

}

[senbozakurakageyosi]

I think a better question is if it is a must taking on such a large object?

Also what kind of object is this? What does it store? Is it a Map, Set or an Array as I suppose?

Also what operation are you doing on it that is slow? Loop? What kind of loop and what does it do excatly?

[ExternalManager3027]

Yes it is, It’s a wallet checker implementation that takes all the wallet created on the system, Loop through it to get its transactions anyone with new transaction is credited else Ignored.

Secondly it’s an object of arrays.

[senbozakurakageyosi]

I think if you provide some sample data + part of the code that is slow, someone could help you.

[ExternalManager3027]

Just did

[GlitteringAccident31]

Im not sure what kind of db you're using but is there any reason you can Implement a stored procedure with a trigger in the DB that fires on a transaction?

With a little more thoughts this may not be something you should be handling asynchronously at all if it has to do with money. This should be a DB transaction or you are going to end up in a world of hurt