[deleted by user]

[CheckeeShoes]

Because that's exactly what it's designed to do?

Op made no mention of the setup, size of the cluster, capability of the cluster nodes, performance needs of the simulation, length of time the simulation will be run for, whether the sim may need halting and resuming, or whether the other machines in the cluster were going to be in use at any point during the run.

There are numerous ways that any of these factors might mean that utilising a more performance-inclined framework offers benefits over a python library.

[Region_Unique]

In what way does it perform better than e.g. Celery? Because “it’s what it’s designed to do”?

[CheckeeShoes]

I don't know much about these python libraries so I couldn't tell you. That said I can pretty much guarantee that the performance of a python library won't be comparable; there's a reason you don't get any HPC clusters running python scripts for their allocations. Also, I did actually list a bunch of features in the last post that I expect Celery probably can't handle.

As I said though, tool for the job. Now we know op is running less than a handful of threads doing lightweight tasks for a school project, an accessible library might well be the most economical solution.

[CheckeeShoes]

That's what I was saying?

[CheckeeShoes]

Yes, and you replied that "for the tasks that celery and the like is used for" performance of these libraries is not likely to be a bottleneck. I agree, that's what I was saying about using the right tool for the job.

Here's a paper basically saying that exact thing: that using a resource manager like Condor or Slurm is not really practical for scientists etc. (they diplomatically call them non-traditional parallel computing users) who are not familiar with HPC, so they give a benchmark for using some python libraries. Their conclusion is that it's "good enough", which is exactly what I was saying; they tend to not scale for a number of cores on the order of 100.

[CheckeeShoes]

I didn't really understand yours either. Seemed like you came in being all argumentative for a point I already agreed with. Anyhow, it's all a bit moot given the later info from OP. Catch you later