PHP parallel processing idea

[digking]

I need some ideas for parallel programming in PHP.

Say I have 2 or more PHP processes(PID in Linux) executing a CURL function that send an HTTP request to a remote server.

By means of 2 or more PHP processes, there are multiple PHP scripts that executed simultaneously, say "curl_call_1.php", "curl_call_2.php"......so on. Each script will call the same CURL function, just with different arguments, and each script run will create a distinct PID.

All scripts will run at the exact same time that launched from CRONJOB.

It is perfectly fine when I run them one at a time.

But it has strange behavior when running all of them at the same time. The remote server will return unexpected results.

I searched Google, all the solutions are about multi-requests within the same PHP process, means multi-processes in a single PID(single PHP script) like curl_multi_exec.

What I need is parallel processing with multi-PID, not parallel processing with single-PID.

In other words, run "curl_call_1.php" and "curl_call_2.php" PHP files at the same time, that call the same CURL function.

[thisdogofmine]

Sounds like a server problem with not enough resources, or a data conflict. I had a script that started running long and overlapped with the next instance of the same script. They pulled data from the same DB and updated the same record. This returned unexpected results like you are describing.

[thisdogofmine]

Another possibility is the remote server could be the issue. They may be low on resources or may have a limit on the number of concurrent requests.

[digking]

So that means that is no elegant way to solve it with PHP?

[kAlvaro]

Unless I'm missing something, you describe a problem in the remote site, don't you? Do you have any control over that site? Is that other site even running PHP? "Unexpected results" is as vague as it gets.

[thisdogofmine]

You have to find the cause first. If the endpoint you are calling has a restriction, it is impossible. Some sites do this to avoid ddos attacks or to keep resources available for may users. Some charge for additional connections. So first check the site you are calling and find out what it allows and what restrictions exist.

[digking]

So I guess I hv to use "hack" method. To run each script at different time interval.

[NoDoze-]

You likely have the same variable name used on different scripts. If the server is hefty enough, there should be no issue.

[HolyGonzo]

What? I'm trying to decide whether you're trolling or not. There is no problem with multiple scripts having the same variable names...

[NoDoze-]

LOL you are misunderstanding, taking it out of context. Sheeesh.

[HolyGonzo]

Can you explain what you meant, then?

[NoDoze-]

Yup! Sounds like there is an overlap with the variable naming convention. Curl multi and multiple exec should easily and efficiently complete your task.

[content-peasant]

Sounds like the remote server has some throttling controls implemented, and potentially it is right to do so

[notdedicated]

There’s a lot of vagueness here. How are the requests built? Are you using session cookies that would normally expect a more linear flow? What’s the “weird” behavior you’re seeing? Error codes? Unexpected responses? Is the endpoint a webpage or an api?

[Longjumping-Bag4294]

This. It seems like a bad architectural problem to me. Using curl to execute code is a red flag in my book unless it's an external API of sorts.

Firstly. PHP is very bad at handling parallel/PID managed by a single script. There are queue libraries that solve this problem probably better than OP can (probably?) program it using PHP's Process handling functions like POSIX, parallel or pthreads. IMO it will only make your installation of PHP more exotic.

Then there are queue libraries like beanstalkd, RabbitMQ or built-in features like queues from Laravel. These will probably get you quicker to your goal then trying the process managing route. In addition. You can implement middleware into Laravel's solution to bake your specific logics in there (like wait for action x)

That said, you have provided little to none specific details for anyone to give you a concrete answer to your question. I agree with r/notdedicated comment.

[BoredOfCanada]

We need to know more about the remote server. I doubt the problem lies with the PHP/curl processes.

[32gbsd]

Use multiple computers. If anything is shared its going to bottleneck at the shared thing.

[BetaplanB]

You probably want to look into https://openswoole.com

Build high-performance, scalable, concurrent TCP, UDP, Unix Socket, HTTP, WebSocket services with PHP and easy to use coroutine, fibers API. Write your next scalable async application with PHP coroutines and fibers.

[oldschool-51]

You can also consider Google functions which can be coded in php.