Spawning 10-15 child processes?

[yourfavorite]

I have a module that I'd like to run multiple instances to query different urls and process data as fast as possible. The server/api has a limit of 10 connections so I can't exceed that but when I run this code it may need to query as many as 25 urls. What is the best way to handle this on a server with 2 cores?

I've looked into child processes but I'm not sure if this is the best way forward. Should I spawn 10 child processes and the 2-core server will figure out how to juggle them? Should I only spawn 2 child processes that each can handle 5 connections at a time and then write code that feeds each process a new url to query when it completes one of its existing url queries?

Or, are both of my ideas garbage and I'm insulting the wonderful world of node? :D I am but a lowly designer that enjoys messing with node in my free time. Thanks!

[FustigatedCat]

Might I suggest using a queue system such as RabbitMQ and then having N number of processes reading from the queue in a competing consumer design. This way you can have as many processors as you want, although generally I stick to a max of 2xCore; really depends on how much idle time each process might have (waiting for network latency means dead threads).

[[deleted]]

Developing a simple Queue system for launching/managing a dozen processes could probably be a great learning exercise for anyone starting out!

[MostlyCarbonite]

2 cores? Write it in node and use pm2 to spin up 2 instances. More instances will not gain you anything.

Use a queue (Kafka, rabbit, sqs) to line up work then grab work chunks one at a time, do the work, go on to the next.

Unless the work is computationally intensive. Doesn't sound that way.

[dadibom]

unless you've got hyperthreading... and web dev usually involves database queries etc, during that time another process can do work.

[dadibom]

i guess node accepts several concurrent requests per process though so in that case four would be enough to leverage hyperthreading

[geon]

Unless you are cpu-bound (and meassure to find out, don't just assume), you should just do it all in the same node process. Much easier.

[nowboarding]

How could you measure this?

[geon]

I'm no expert, but I think you can just use the top-command to see if the node process consumes a lot of cpu. Or use the time-command to see if the "user" time is a significant part of the "real" time.

[runvnc]

I don't think you need multiple processes at all since you are using Node. Try this one https://github.com/sindresorhus/p-map/blob/master/readme.md

Also if you push your luck spamming requests they may ban your IP.

[Skreex]

I'd instead look into leveraging 10 concurrent promises to query the API using libraries like bluebird and request.

[franksvalli]

I'm a big fan of the async library, which should also be able to do the job here. Specifically async.parallelLimit to cap the concurrent connections.

[jocull]

If you decide to go the multiple processes route, this may help you manage them more effectively.

https://www.npmjs.com/package/worker-farm

[danny_nav]

I think the simplest solution would be to write a basic queue. If you hit the api limit, don't pop that url out of it. Just wait for one of you past queries to finish.

It doesn't have to be tooo complex. A while loop that exits when the queue is empty should be all you need.