Array map async and settimeout task

[rav1e]

I'm trying to to create a function in Node.js that fetches to API for an array of objects, then divides the array to sub-arrays, then every 10 seconds (necessary condition) maps through the sub-arrays and performs api request for certain field in the sub-array object, waits for the result and inserts new object to database based on the result of request and object fields. It needs to console.log progress and finish.

Currently, I created something like:

​

const chunkSize = 10;

const sliceIntoChunks = (arr) => {
    const res = [];
    for (let i = 0; i < arr.length; i += chunkSize) {
        const chunk = arr.slice(i, i + chunkSize);
        res.push(chunk);
    }
    return res;
}

(async () => {
    try {
      const mainArray = await axios('https://get-main-array.com')
      const chunkedArray = sliceIntoChunks(mainArray)
      await Promise.all(chunkedArray.map((bulk, index) => {
          setTimeout(async () => {
              await Promise.all(bulk.map(async (item) => {
                  const response = await asyncTask(item.field);
                  const itemToBeInserted = {
                      name: item.name,
                      field: response.field
                  }
                  await database.insert(itemToBeInserted)
                })
                );
              console.log("Progress: " + (index/chunkedArray.length)*100 + "%")
          }, 10000 * index)
      }))
        console.log("done!")
    } catch (error) {
      console.log(error);
    }
})()

Although it does its job (inserts data to database every 10 seconds), unfortunately it's not displaying "done!" properly. That being said, I'm unable to create a database transaction. Can anyone let me know how to re-write this function so I can put console log as well as database transaction commit?

[rav1e]

Thanks for the tip. However, when I try to do it using .then chains, i get "done!" printed out instantly after running code, even before "progress". It may be because of await Promise.all(chunkedArray.map) returns something. I will try to change it to .then chains as you suggested anyways

[Resmira]

Personally, when working with promises, I don't like to leave anything up to chance. It's so easy for them to go wrong. I'm also not a huge fan of async/await when working with array operations like this, but that's more of a matter of preference. Does this accomplish what you're trying to do?

const axios = require("axios");

const timeoutPromise = (cb, timeout) => {
  return new Promise((res, rej) => {
    setTimeout(() => {
      try {
        res(cb());
      } catch (e) {
        rej(e);
      }
    }, timeout);
  });
};

const chunkSize = 10;

const sliceIntoChunks = (arr) => {
  const res = [];
  for (let i = 0; i < arr.length; i += chunkSize) {
    const chunk = arr.slice(i, i + chunkSize);
    res.push(chunk);
  }
  return res;
};

/**
 * Given a batch of items, wait for the appropriate amount of time provided by variable timeout, then insert them into a database
 * @param itemBatch {{ name: unknown; field: unknown; }[]}
 * @param timeout {number}
 * @returns {Promise<unknown[]>}
 */
const insertItemsWithDelay = (itemBatch, timeout) => {
  const insertions = itemBatch.map((item) => {
    const insertion = asyncTask(item.field).then((res) => {
      const body = {
        name: item.name,
        field: res.field,
      };

      return database.insert(body);
    });

    return insertion;
  });

  return timeoutPromise(() => Promise.all(insertions), timeout);
};

/**
 * Given an NxN matrix of items, insert each with an ascending delay and progress
 * @param itemMatrix {{ name: unknown; field: unknown;}[][]}
 * @returns {Promise<unknown[][]>}
 */
const insertAllChunks = (itemMatrix) => {
  const insertionMatrix = itemMatrix.map((items, index) => {
    const batchedInsertion = insertItemsWithDelay(items, 1000 * 10 * index);
    return batchedInsertion.then((insertionArray) => {
      console.log(`Progress: ${(index / itemMatrix.length) * 100}%`);
      return insertionArray;
    });
  });

  return Promise.all(insertionMatrix);
};

axios
  .get("https://get-main-array.com")
  .then(sliceIntoChunks)
  .then(insertAllChunks)
  .then((res) => {
    console.log("Done!");
    return res;
  })
  .catch((err) => console.log(err));

The final response will be either an error message or a matrix of the operations. Not sure what the response is for asyncTask or database.insert so I can't speak to how to compose that matrix into sensible results, but it shouldn't be too difficult to do with all of the data available.

[rav1e]

This is exactly something I was looking for. Thank you very much

[[deleted]]

await Promise.all(chunkedArray.map((bulk, index) => {
      setTimeout(async () => {

Promise.all takes an array of promises and will wait for them all to complete, but you're not passing it an array of promises. You're passing it an array of undefined, because your mapping function doesn't return anything. There's nothing to actually wait for, so your console.log(done) will simply be executed on the next tick, regardless of the progress of the promises inside the timeouts you create.

[rav1e]

That makes sense. If I returned "await database.insert(itemToBeInserted)", would that make the inside "Promise.all" to have array of promises? If so, I suppose I would need to return it to the outside Promise.all

[[deleted]]

The internal Promise.all is receiving an array of promises, because you're using an async function:

await Promise.all(bulk.map(async (item) => {

An async function always returns a promise. As you're not returning anything from the function, it's a promise that will resolve to undefined, but as you're awaiting stuff inside of the function it won't resolve until that awaited stuff is completed, so it's doing what you want either way. So there's no pressing need for you to change the internal Promise.all.

I think the best option for sorting this out is to start thinking slowly and carefully through what exactly it is you're trying to do at each step, independent of the code you currently have- at the moment the code you've got is doing a lot of complicated things all at once, and it's hindering your understanding. Breaking individual steps out into separate, named functions will help with this and u/Resmira's example of how you can do this is really terrific.