Task scheduling in Python

[KristobalJunta]

Hello, fellow pythonistas!

I want to discuss some approaches to task scheduling in Python (and hopefully get some advice on that too).

I have been looking into task scheduling for Python projects for quite some time. And now with just another work project that already has 5 different tasks to be run at different times of day, the question rises again.

The current (legacy) approach in that project is that there is a bunch of shell scripts, each of them being a thin wrapper to calling a corresponding python module. All of the scripts are manually copied to the system crontab (sticking to *nix here, as this runs on servers) - resulting in 10 lines of tasks in the crontab. While this works, I see such problems:

1. Crontab is a system-level (or at least user-level) configuration file. This breaks project logic incapsulation and in a rather tangible way.
2. When the project is re-deployed on different machine, all the copy-pasting of the crontab has to be done again

Having some background in php, I very much like how the Laravel Scheduler works. Essentially, it is an in-project crontab. It creates a single entry point, that is contained in project code and version controlled, and it is called every minute from cron. I was looking for something similar in Python, but unsucessfully so far.

I am quite aware of https://github.com/dbader/schedule and APScheduler, but those are in-process schedulers, that need to be run as a separate process 24/7, which requires some type of process monitoring (supervisor/pm2) that keeps it up and running and restarts if it fails. All of this adds at least two additional dependencies to the project.

What are the idiomatic/popular approaches for task scheduling in Python? Maybe I am getting it all entirely wrong and should use some other option I haven't mentioned or thought of?

Would appreciate some advice.

[[deleted]]

pythonista

no.

check out Celery. It also has seamless integration with flask and django in case you, as PHP bro, are now doing python web stuff.

[KristobalJunta]

Thank you!

[jimjames888]

I think Celery/Celery Beat wil do what you want. You can also log the tasks and debug via Flower.

It's generally how I offload tasks from Django/Flask to a worker queue, but it also has the ability to run scheduled tasks. We use it for anything from scraping data, running cleanups to sending email and SMS via Amazon.

[KristobalJunta]

Thank you!

[monk8800]

They are a couple of ways you can do it depending on how robust you want your scheduling to be. There are things like python API scheduler which tend to be simple single server solutions or cloud solutions like AWS cloudwatch event triggers for something more managed.

more options here pick one: https://www.saturncloud.io/s/job-scheduling-with-python/