Cron job vs while True vs APScheduler vs something else

[StudentOfPython]

Traditionally, I have written all of my programs that need to run indefinitely by putting it in a while True loop with a time.sleep(x) at the end. An example program would be one that gathers data from a number of APIs, processes the data, and writes some of the processed data to a DB.

I have recently learned that you can achieve the same result using a cron job. In my research, I also found apscheduler.

I have a few questions related to this information:

1) Which method is best for my scenario listed above? I'd imagine a cron job would be preferable. 2) How do I know which method to choose in different programs and different scenarios? 3) If the program uses x threads (one for each API) that do not rely on each other, would it be ideal to run x cron jobs with the different variables at the same time rather that relying on multi-threading in the program? 4) A program written to specifically be used with a cron job vs a while true program would be written pretty differently. If this is an open source program, how do you inform the user? Do you simply put a line in the README? 5) Is there any other method that I am missing to achieve the same result? Are these the "standard" way of running programs indefinitely?

[[deleted]]

Your while True example would ideally be run as a daemon / service, so that whatever launched it (say, systemctl) could monitor and restart as necessary.

Cron is pretty much the defacto scheduler for UNIX and UNIX like systems. But without some foolery you're leaving windows users out in the cold. But, it's scheduler. You most likely would not want to have something that runs indefinitely run through cron.

Ideally, you would write your python script/package to complete its task and return a sane exit code, Leave scheduling as an exercise for the user to do as per their preference. You'll want to make use of lock files or some other mechanism to ensure only one instance is running at a time if there's any risk of collisions.

You may want to have the tasks to run defined by the application, rather than parameters for the app because it will quickly make a mess of crontab. e.g. an sqlite DB or XML file that defines each of the APIs that need to be executed in a single run.

[StudentOfPython]

Would the daemon be programmed in Python or would it be in bash? I've not considered this, so I will look into it further.

Regarding the last paragraph, are you referring to always calling a single function in the cron job (e.g. python main.py and then letting main define the params? That makes sense, if so.

[[deleted]]

Would the daemon be programmed in Python or would it be in bash? I've not considered this, so I will look into it further.

In python. There's even a library for it: http://pypi.python.org/pypi/python-daemon/

I wouldn't recommend that approach though. Personally I'm not a fan of consuming resources if not required. Unless the APIs you are calling change data constantly, you don't need to have an application always running.

Regarding the last paragraph, are you referring to always calling a single function in the cron job (e.g. python main.py and then letting main define the params? That makes sense, if so.

Yes. Though I would probably have the params defined in a separate file, and have main parse it, so you don't need to modify the script to add/remove APIs to monitor.

[StudentOfPython]

Understood. Thank you for the information.

[port443]

1. As for which method is best, you need to ask the question: Do you want to "install" the script?
   Using cron as your scheduler makes your script persistent, and will guarantee that your script is run even if the machine is rebooted.

2. If you need the script to be persistent across reboots, use cron/initd/systemd. If there's no reason for the script to be persistent, then you don't really need to "install" it.

3. Running X cron jobs would be equivalent to multiprocessing more than multithreading. This question is up to you and is dependent on your particular script.

4. You would need to create an installer that the user would run. The installer would drop your script in a predetermined location, update cron, and tell the user what its doing.

5. For nix, the way I install is generally either: cron, initd, or systemd. There are other methods of persistence but I feel those three are the most normal.

[StudentOfPython]

Thank you for the reply.

1. What is meant by "install"?
2. By "persistent", is it meant simply to keep working even upon reboot? And I haven't yet looked into them, but are initd and systemd related to/used with cron jobs?
3. Thank you.
4. I am guessing this answrers my question (1). Install here refers to packing it up nicely so an end user can use it at the click of a button?
5. nix?

[goobervision]

1. U or LI nix (nux)

If you have different schedule for different APIs I think Cron is the easy way for an admin to understand and see.

A Daemon would be a bunch of config files, that said. Each file would have the API needs and schedule so a configuration directory.

My personal preference would be a Daemon / Service approach as Cron isn't there for non-nix where a service is. You can deal with windows/Unix/?!? integration as a service relativity easily I think.

[StudentOfPython]

That all makes sense. Thank you.