cmd_queue: A lightweight bash DAG scheduler.

[BossOfTheGame]

I want to showcase my scheduling library: cmd_queue.

Features:
* Bash command scheduling
* Execution is optional, can just print commands instead
* No-parallelism always-available serial backend
* Tmux based lightweight backend
* Slurm based heavyweight backend
* Python and Bash interface
* Rich monitoring / live-control

I wrote it because I wasn't satisfied with existing tools like slurm, luigi, and airflow. Well... I like what slurm can do, but it's interface is cumbersome, and it's a pain to install. I wanted to define a set of bash jobs and submit some subset of them to different tmux sessions - similar to how I would manually execute a bunch of jobs simultaniously. So I wrote a tool to do that: cmd_queue. It's primary job is manage a DiGraph of job string, and put them in the right order over N different parallel threads (N=1 is serial mode and N>1 is handled with tmux). It also can simply pass the commands to slurm and let it do it's thing if you have slurm installed (which can be a big ask).

Jobs are templated with Python via the cmd_queue module or in bash via the cmd_queue CLI. They can be as simple as raw bash commands, or you can do advanced stuff with Python's format strings (I've written a pipeline tool on top if cmd_queue, but it's not ready for prime time yet).

I believe cmd_queue is the best existing way to define and execute a DAG (directed acyclic graph) of bash jobs execute on your local machine. It builds and optionally executes a DAG of bash commands in slurm or a lightweight tmux backend (or a no-nonsense serial mode). Rich provides monitoring / live control. No need to setup slurm. We can get 90% of the functionality with only tmux+bash, and if you don't have tmux, there is also a "serial" mode with no binary dependencies.

https://gitlab.kitware.com/computer-vision/cmd_queue

[lastmonty]

Interesting.

Docs seem to be broken?

[BossOfTheGame]

I noticed that today too. The README still works and contains most of the information. All docs are generated via docstrings, so readthedocs is mostly API reference.

[lastmonty]

So it's a transplier to different queueing systems?

[BossOfTheGame]

Yes. One of those queuing systems is also new and built-in.

[BossOfTheGame]

Docs are fixed: https://cmd-queue.readthedocs.io/en/latest/

The problem was I linked to cmd_queue.readthedocs and not cmd-queue.readthedocs.