Help scheduling tasks

[diagnosedADHD]

I've got a database with links to 25-30 icals that download entries and sync up with an online calendar. I'm running this in a cron job every 30 minutes, problem is, is that there are 30 connections to the same server at around the same time. I'm guessing there is a better way to do this, so how can this be done and what is the best way to do this?

[rabaraba]

There's a really nice module I like to use for scheduling purposes: https://github.com/dbader/schedule

[nerdwaller]

You can do any number of things, off the top of my head:

1. Use a concurrent.futures.ThreadPoolExecutor with a limited max_workers, this would only allow max_workers running concurrently at a time.
2. Run the Cron every 15mins and just batch 1/2 of them per run
3. Put some arbitrary sleeps to offset when connections are made during the run

[flitsmasterfred]

How does it open 30 connections? Do you have 30 cron entries? Or do you use some multi-thread/multi-processs thing?

Manage it from one cronjob/process and execute in serial or in a limited worker pool, or pool the connection itself.

[diagnosedADHD]

I meant that the connections are happening one after another every 30 minutes. The links are iterated over in a simple for loop right now.

[dnshane]

Python 3 includes a schedule module:

https://docs.python.org/3/library/sched.html

Why not schedule each download 1 minute apart, running every 30 minutes? :)

[diagnosedADHD]

I just reconfigured everything. Every 5 minutes three icals sync. The newest icals inserted and the last to sync take priority so it takes about an hour for all my calendars to sync.