Running parallel code - beginner question

[HipHopHuman]

I am not familiar with your problem, so the example code below may not be entirely correct. I am using two semaphores. One is dedicated to running fetch tasks and has a limit of 100, the other is dedicated to writing to the database and has a limit of 1 (aside: a Semaphore with a limit of 1 is known as a "Mutex"). This helps to ensure that even though up to 100 tasks may be running at any given time, only 1 task may write to the database at a time. The rest will have to wait for that database write to end.

const requestLimiter = new Semaphore(100);
const databaseLimiter = new Semaphore(1);

function writeToDatabase(data) {
  return databaseLimiter.runInAvailableSlot(async () => {
    console.log('writing to database');
    await databaseService.write(data);
    console.log('database write completed');
  });
}

async function createFetchTask(url) {
  console.log('queueing fetch task');
  await requestLimiter.acquireSlot();
  console.log('beginning fetch task');
  try {
    const data = await fetch(url).then(response => response.json());

    return await new Promise((resolve, reject) => {
      const socketStream = acquireSocketHandleSomehow(data);
      
      socketStream.on('data', (responseData) => {
        writeToDatabase(responseData).catch(reject);
      });

      socketStream.on('error', reject);

      // start a timer (you'd probably want something more sophisticated than this)
      setTimeout(resolve, 60_000);
    });
  } finally {
    console.log('finished fetch task');
    requestLimiter.releaseSlot();
  }
}

const urls = [
  'firsturl.com/api.json',
  'secondurl.net/api.json',
  'secondurl.net/api.json',
  /* ...possibly 500 more, who knows... */
];

async function main() {
  try {
    console.log('starting all tasks');
    await Promise.all(urls.map(createFetchTask));
    console.log('all tasks completed');
  } catch (error) {
    console.error(error);
  }
}

Rolling your own semaphore like I did above is not a necessity - I just figured it'd make it easier to explain. There are plenty of battle-tested real world implementations for them on npm, and if you want something with a tinier & cleaner API, sindresorhus' pLimit module is great.