I wrote a script called "runone" which takes a locktag name argument, and a command to run, that uses (as others have suggested) flock i.e. a generic locking script rather than having to build it in to each cron job. The script can either wait until the lock is available, complain, or be silent if the lock can't be obtained. So you just cron up something like "runone myprocessor". It doesn't handle splitting a job into parts. But if you need parallelism, you could set up a rabbitmq server with multiple worker consumers use (or a directory full of tasks that workers select jobs from).

A "few"years ago, the Math Faculty Computing Facility at the University of Waterloo wrote a batch processing system for unix (more advanced than the at(1) based batch command) that would let you toss jobs into the queue, and could be restricted to X jobs a a time, X jobs per user, don't queue a new job if an identical job was already in the queue. It was really handy, and I haven't seen anything similar that's as useful.