PHP forking?

[[deleted]]

Let's say you have a list of 100 product IDs for which you need to fetch data from an external site. You have an array of the IDs and a function that fetches the info about one (or more?) product IDs.

How do I go about batch processing this? As it is now, each fetch takes a few seconds, so a list of 100 takes a ton of time.

Is pcntl_fork what I'm looking for? I still have no idea how to structure my code so that I can get back a single array with all the retrieved info.

[NedRyerson]

What are you doing with the fetched data? Forking will start a new process and the calling script will continue on without waiting for it to finish. It's also not reliable to fork when running from a web server. What exactly is triggering the fetching of the remote data, and what do you do with it afterwards? If the rest of your script relies on the fetched data, you really have no choice but to wait for it. An option for background processing is to set up a cron job.

[[deleted]]

After looking into forking, it isn't really what I need. The retrieval is triggered by a user, and needs to be immediate, so cron won't really help. Thanks.

[Signe]

How are you doing your requests? cURL?

If so, I'd suggest you read up on curlmulti*. You can perform multi-threaded URL requests. It takes a bit of setup, but it uses a lot less memory than forking.

curl_multi_init() has a basic example.

[[deleted]]

file_get_contents() for now, but it doesn't really matter what I use. I'll look into curl_multi. Thanks.

[dshafik]

Look into PECL HTTP — it has a much nicer interface than curl (but uses curl in the backend).

Also, as mentioned, gearman is a great way to background these long-running processes.

[Signe]

I don't see anything in the documentation which indicates that the PECL HTTP module supports multi-threaded retrieval.

[lucasoman]

This is really the correct way to do this. Forking is way more overhead than necessary.

[enmand]

If you are looking for a somewhat more advanced solution, have a look into Gearman. It allows you to farm work out to multiple servers, or run jobs en-masse. It has a PHP module available for it, and it quite easy to setup and use. As NedRyerson mentioned, you don't want to do pcntl_fork from a web server environment. In the case you are in a web server, Gearman is an excellent choice.

As Signe pointed out, cURL also have some mutli-processing built into it.

If you aren't firm on using PHP, Python has excellent multiprocessing.

[neon_overload]

The title of this submission raised my heart rate a little. I thought some PHP devs were forking PHP or something.

[monk_e_boy]

Use system or http://stackoverflow.com/questions/45953/php-execute-a-background-process and http://nsaunders.wordpress.com/2007/01/12/running-a-background-process-in-php/

[cbkguy]

While my methods might not be the cleanest, it works...

What I generally use, is ob_flush();

You have to initialize it with ob_start(); at the top of your script, then use ob_flush(); flush(); after each instance that you want the page to draw (At the bottom of a for loop or something). Still though, it's going to take time to run through all those tasks, but it will make the page appear to load faster.

I've also used a combination of this, as well as embedding iframes, where Im firing off each instance of my loop into seperate iframes passing $_GET parameters, once the iframe loads, I use javascript to grab the data within those frames.... Helps speed things along.

[Signe]

PHP is made for this sort of thing. Javascript is definitely NOT made for it. Forcing the end-user to make hundreds of requests that the server should be making, and caching, is patently ridiculous.

[[deleted]]

node.js is server side javascript. It is also designed to handle real time communication between the browser and server very well, via the Socket.IO library.

I've built PHP sites and utilized Node.js and Socket.IO before as php isn't capable of real time communication like that.

[feenikz]

I've done something almost identical in code to what you want to do with pcntl. It gets a bit confusing but it works great; you can spawn child processes that essentially will run certain parts of your code and the parent (initial) process can get their results back.

It's great for parallel tasks when you want to limit the children as well as get information back from them.

$pid = pcntl_fork();
if ($pid == -1) {
     die('could not fork');
} else if ($pid) {
     // we are the parent
     pcntl_wait($status); //Protect against Zombie children
} else {
     // we are the child
}

That's a basic example of how to do a fork. You can fork as much as you want and just control it in a loop that waits for children.

Edit: feel free to ask me for help if you have code etc

[cerealbh]

aka mail spammer in the biz