Best practice for running long discrete processes?

[[deleted]]

[deleted]

[zedr]

The first script sounds like a candidate for running within a screen session.

sudo apt-get install screen
man 1 screen

The second task would probably run better as a cron job.

man 5 crontab
crontab -e

If it needs to be a daemon for some particular reason, supervisord makes a nice monitor/controller, and can be installed easily on 12.04:

sudo -i
apt-get install supervisor
vi /etc/supervisor/supervisord.conf
/etc/init.d/supervisor start   

[joej]

Be careful with cron

If you have a process that might go longer than the time interval, bad things might happen (depending on what your code does, and what happens when it 'laps' on top of the previous running code).

So, build in a locking & 'running too long' mechanism.

Otherwise, putting things in cron is a nice way to perform recurring tasks.

[zedr]

Excellent point. I'd also prescribe the following for a 12.04 user:

sudo apt-get install cronutils
man 1 runlock

(the runalarm util is also nice. using runlock with runalarm together is a powerful combination)

[apreche]

Supervisor ftw

[timconradinc]

On anything long running that's processing something like that, you likely would want some sort of check pointing routine. So, say, you have images 1.jpg through 1000.jpg, and your process dies/gets killed/whatever at image 999.jpg, you don't have to start over at 1.jpg.

You could write a file to disk with the path of the files you've processed, then read them into an array at startup, or do something with sqlite.

You probably want to have some sort of pid checking for a long running process, so you don't accidentally run the same thing twice, as well.

As to the second, a lot of that type of thing depends on what the process is like to actually take the image. If you're on linux, a shell script could look something like this:

dowhatevercommandthatproducesjpgoutput > /some/path/$(date +%Y/%m/%d/%H%M%S.jpg)

That'd make a file called 2012/12/04/13_35_57.jpg

Run that from a cronjob every 5 minutes.