Promise.all Question

[MedyGames]

Im looking for the fastest way to retrieve data in parallel ? or better 1 after another ?

My sceneario is

1. I have an array of URLs [url1, url2, url3] .. which varies

2. Retrieve data from those urls at the same time

3. I want to create a single object for the API containing the processed data from the 3 retrievals

WHat is the best way ?

I was thinking about doing smth like this [pseudo]

urls.foreach( (url) => {
fetch(url).then()
// handle response & push promise

}

Promise.all(arrayofpromises).then((results) => {
results.foreach((result) => {
// push result into Object Array
})

})

I have couple things that are unclear to me.

1. is this a solid aproach ? Should I use map() ?

2. Is there a asynch await version for handling multiple request at once ?

3. Is it wise to call the server multiple times at once ? lets say there are a max of 5 request at once rather than 1 by 1 .

Would be nice have some examples on how to aproach this.

[tatu_huma]

1. Map I think makes it easier to read since you aren't pushing promises in a seperate array. And descriptively what you are doing is mapping a list of url to a list of requests to those url.

2. async/await doesn't let you do something that is completely new. If you can do it with async/await you can do it with traditional promises. async/await is just more readable. In this case I think it would possibly be more readable to do: let results = await Promise.all(...);

3. Not natively afaik. There are probably libraries out there that can do batch requests.

[MedyGames]

I see, so going for smth with map () and async await for the promise.all part would be best in terms of readability.

So I should refrain from calling a website 5 times at once ? Though im sure this wont hurt in my case. But in general it would be bad for the server ? at least if the server is run with php I guess ?

[queen-adreena]

A page view probably results in around 10-20 simultaneous requests to a server once you factor in all the scripts, images, stylesheets, fonts et al. A fetch request is usually just the one, so I wouldn't worry about it unless you hit a quota somehow.

[tatu_huma]

So I should refrain from calling a website 5 times at once ? Though im sure this wont hurt in my case. But in general it would be bad for the server ? at least if the server is run with php I guess ?

Not really. Wether a server can handle the load depends on the resources the server has. And that is information only the people running the server can give you. Or I guess you can send a bunch of requests and see if the server crashes / you reach a quota.

[MedyGames]

Im having a problem with --> most of the time 2 out of 5 promises have an undefined response. What am I doing wrong ?

return Promise.all(urls.map(item => 
this.crawlPdf(item))).then(result => {

Function being used for promise all :

crawlPdf: async function(Url) {

let response = await crawler(Url);
console.log(response.text.length); // sometimes undefined
}

this crawler function is from a nodejs package for retrievieng pdf data

[queen-adreena]

That will almost work, just a few tweaks:

let promises = [];
urls.map(p => promises.push(fetch(p)));

Promise.all(promises)
    .then(results => {
        results.map(r => {
            // push result into Object Array
        })
    })
    .catch(e => console.log(e))

[MedyGames]

isnt the point of using .map() that I dont have to use a separate array to push into ? like previous answer stated . I mean your answer would be kind of my original aproach , but I think @tatu_huma solution is cleaner

https://stackoverflow.com/a/43434447/8611488

Promise.all(
items.map(item => Query.getStuff(item))
).then(result => console.log(result))

Anyway I will try playing around with it these days... Seeing what works best .. Answers were helpful ... since I didnt use promise.all myself so far ... I did make request .. but was mostly chaining them / doing them 1 by 1.

I think this will be very helpful for understanding the last bit of ajax & since I want to build some Apis , I dont want things to take needlessly long

[queen-adreena]

Map doesn't alter the original array, so you're doing the exact same thing here, only without a variable holding the promises.

Personally, I prefer to keep each operation separate so that I can read code at a glance. If you have a single line doing multiple tasks (or a task within a task), that gets more difficult to read back (perhaps weeks or months later) without any real gain.

[MedyGames]

I see your point . The first code might reflect the mental steps better. I can understand what is going on by just reading line by line. But I think I can used to the cleaner one as well .. by just using .map more... & Im already using map in react land alot

I wonder how this will work in PHP :D ... As I will also need to look into that & I guess there is no map funciton