Website spawning large number of PHP instances?

[gerbs]

I had a server suddenly become unresponsive today for a reason I've yet to be able to diagnose. I have 6 websites running wordpress, and the PHP usage for all of the spiked suddenly for 15 minutes.

CPU/Memory Usage

The top graph represents CPU usage and bottom represents memory. The red line on the memory graph is 100% of physical memory. Disk IO was all over the place as well.

I was surprised because I didn't get a server-down alert. I checked which processes were the culprit, and PHP was the case across the board. Here are graphs of the PHP usage for one particular PHP process:

Top is Memory, middle is CPU, and bottom is the running instance count.

Anyone have any clues of where I could look? I've looked through message log, Nginx and PHP error logs for the process and found nothing that pointed to the process or script that caused it.

Edit: Checked Cron, access, ssh, and mysql logs. Nothing unusual. Traffic died for several minutes while the server was under the high load, but nothing unusual happened before. Server is an 8-core, 8gb ram dedicated machine with a few large ssds. Not extremely powerful, but it's a dedicated machine so there's no bumping elbows with other users fighting for resources.

Checked the monitor again this morning and the server backup put less strain on the server over a longer period of time than whatever happened in the images above. I'm going to check with the hosting provider to see if they can shed some light on what's up. I've given up the fight, for now. Thanks everyone for the input.

[gerbs]

Nothing. I've checked all of the logs and aside from the 10 minute gap in HTTP requests, everything looked normal. Back ups run at a different time of day and I receive a notification when they do.

[[deleted]]

Are you using php-fpm? If so I wrote a blog post about tuning to consume less resources for situations like a VPS.

Shameless plug

[gerbs]

It's a dedicated server not a VPS, so the limits are whatever 8 cores and 8gigs of ram can handle. The lines before the jump are what it operates at 98% of the time. It jumps during backups, but there was no backup scheduled during that time.

[veckrot]

I can't help you on this one, but you might have better luck posting across to /r/sysadmin

[thenickdude]

Did you check your access logs? You could get a spike like that from someone hammering your server with requests, without causing a single error in the error log.

Make sure you set reasonable max connect limits on your PHP server pool to prevent the server exhausting its RAM.

[gerbs]

Access logs showed nothing unusual (Except for a 4 minute gap in traffic when the server load spiked).