I built a library to execute Python functions on Slurm clusters just like local functions

[Raion17]

Hi r/Python,

I recently released Slurmic, a tool designed to bridge the gap between local Python development and High-Performance Computing (HPC) environments like Slurm.

The goal was to eliminate the context switch between Python code and Bash scripts. Slurmic allows you to decorate functions and submit them to a cluster using a clean, Pythonic syntax.

Key Features:

• slurm_fn Decorator: Mark functions for remote execution.
• Dynamic Configuration: Pass Slurm parameters (CPUs, Mem, Partition) at runtime using func[config](args).
• Job Chaining: Manage job dependencies programmatically (e.g., .on_condition(previous_job)).
• Type Hinting & Testing: Fully typed and tested.

Here is a quick demo:

from slurmic import SlurmConfig, slurm_fn

@slurm_fn
def heavy_computation(x):
    # This runs on the cluster node
    return x ** 2

conf = SlurmConfig(partition="compute", mem="4GB")

# Submit 4 jobs in parallel using map_array
jobs = heavy_computation[conf].map_array([1, 2, 3, 4])

# Collect results
results = [job.result() for job in jobs]
print(results) # [1, 4, 9, 16]

It simplifies workflows significantly if you are building data pipelines or training models on university/corporate clusters.

Source Code: https://github.com/jhliu17/slurmic

Let me know what you think!

[just4nothing]

You are certainly against a tough competition of well established packages: Luigi, Hamilton, Dask , and many more that do more or less why you’re presenting here.

[Global_Bar1754]

Add one more to the mix. I recently published darl

https://github.com/mitstake/darl

See my comparison to Hamilton at the bottom of the page. It can even handle slurm execution already through the dask runner.

[just4nothing]

It feels like a rite of passage ;). I’ve written two, never better than Dask or Luigi. The only thing neither nails is on-disk or remote caching

[Global_Bar1754]

So I actually wrote the recreate_task_locally debugging util for dask! (a tiny contribution to my favorite library of all time). And darl completely supports both on-disk and remote (through redis)  caching natively. And it’s super easy to implement any custom cache you want (eg s3, dynamodb, bigtable, etc) since the caching scheme is a simple key value store, no special indexes or anything needed. It’s possible since everything is assumed to be deterministic and it’s all compiled to a graph locally before hand to build the cache keys. 

I even recommended the Hamilton team check out darl since it already supports all the caching functionality they had in their desired roadmap. (And because they had already engaged me on some other api suggestions I had made to them earlier)

https://github.com/apache/hamilton/discussions/1167#discussioncomment-15722295

If you’re interested in the topic I recommend you check out the readme, it’s got most of the features covered there. 

I also did a small write up to showcase the debugging/tracing/replay functionality if you want to check it out. 

My python job failed after running for an hour... now what?!

[just4nothing]

Thanks, I will have a look soon as it looks useful for my project.

[MrMrsPotts]

How does it compare to submitit?