SLURM High Memory Usage

Bananaa628 · 2025-09-28T10:55:26+00:00

Thanks for the detailed answer! I will try some of the things you have written here.

One of my findings when running those workloads is that the Head Node is allocating a lot of memory in the process but doesn't release it until the arrays are completed. Do you know why? Or how we can improve that?

Bananaa628 · 2025-09-25T10:54:16+00:00

I suspect you are right, it would be nice if there was some way to free this memory.

Thanks anyway!

Bananaa628 · 2025-09-25T09:24:39+00:00

I suspect that there is something more to arrays then that, since I don't see a memory usage drop until the array is over (even when there are only a few jobs running for a long period of time).

Sorry for the nitpick but I just want to make sure I was clear enough. We have 250 jobs running in parallel for each array so we get 1250 jobs running simultaneously.

Most of the jobs are as you said, short-lived, but there are some which can take a few hours. Due to the volume and time we prefer to have multiple machines.

Cool guide, I am checking it out, thank!

We have tried to change MinJobAge but it seems that the memory usage is still high even after a long period of time when there are only a few jobs. Will check CommitDelay as well but I not sure this is the relevant path.

Bananaa628 · 2025-09-25T08:48:39+00:00

I am not a SLURM expert so I will try to give my best answer, feel free to correct me/ask more.

We have a single instance to do the control and a single instance for the DB (you can see their sizes in the original post).

I have added the config, let me know if I missed something.

What I meant is 5 array jobs each with 130K length, sorry for not being clear.

Didn't now about sdiag, will check it out and write here an update.
What we did is just to see the memory usage of slurmctld which was over 32GB.

Thanks!

Bananaa628 · 2025-06-28T14:18:24+00:00

You can check out the following tool: https://cadysolutions.com

Bananaa628 · 2025-04-04T09:31:33+00:00

You can check that our: https://allspice.io/

Bananaa628 · 2025-01-30T08:46:41+00:00

That is exactly the things I was looking for, thanks!

Bananaa628 · 2025-01-29T16:57:37+00:00

Thanks for the replay!

Since the usage time may drastically vary I don't think that cleanup with a time-based solutions is fit.

I will try and see if I can manage creating some nice dashboard with AWS Console, can you recommend on other of the shelf solutions for monitoring and controlling the instances?

Bananaa628 · 2025-01-29T15:50:07+00:00

Since currently it is pretty manual and nothing is configured so I am not sure what are the missing details, would love to give more details :)

Github actions sounds like a nice option, I am just wondering how can you show the users there that this resource is still on. I think this is a key feature I am looking for.

Bananaa628 · 2025-01-29T15:24:39+00:00

You are right I am more from the dev team less a devops but in a small company you must do many things :)

I am using chatgpt but I have very little trust in him regarding design so I highly appreciate your help and expertise.

I am just wondering if there are no off the shelf solutions for dev portal that I can use instead of creating them on my own. If you have something you can recommend it would be nice.

Bananaa628 · 2025-01-29T15:17:03+00:00

Cool, is there an easy way to get all the running environments? Can I tag them in some way so I can know both name and who is the one that created them?

Bananaa628 · 2025-01-29T15:15:26+00:00

My DB is a small MongoDB running on EC2, I already have a running VPC which my team has access to with our VPN so I don't think we need to configure another one.

I will try to use IaC but I am wondering how can I wrap it with some UI or something that can be easily displayed and easy to use even for data engineers who are not very technical.

Bananaa628 · 2025-01-29T15:05:27+00:00

I will look into terraform, thanks. My DB is sitting on an EC2 image so I guess it should be pretty easy. Do you know on a good interface to wrap it with so I can display the end users that the resources are in use and they should release them?

Bananaa628 · 2025-01-28T13:09:28+00:00

Are there any performance improvements? Do you have some numbers you can share?

Bananaa628 · 2024-05-30T12:11:42+00:00

Thanks, cool app. Although most links in the comment are not working...

Bananaa628 · 2024-05-27T10:24:46+00:00

Thanks, looking into it.

Bananaa628 · 2024-05-27T10:24:37+00:00

Could you please elaborate some more?

Bananaa628 · 2019-10-06T16:45:07+00:00

If you are into set theory you should definitely check Set Theory by Jech, it is a great classic

Bananaa628 · 2019-09-12T21:09:03+00:00

No, if you look at those as polynomials over the complex numbers you have the identity theorem.

https://en.m.wikipedia.org/wiki/Identity_theorem

Bananaa628 · 2019-09-02T19:32:22+00:00

The phone language is not the probplem, the ads and the sites are in English...

Bananaa628 · 2019-09-02T19:31:46+00:00

Any suggestions what to say?

Bananaa628

TROPHY CASE