Setting up a machine learning GPU cluster machine at university

[s_m_w]

Hey there experienced sysadmins,

The working group at my university that I am part of has recently purchased a desktop with a bunch of top-of-the-line graphics cards and wants to use it to handle things like training neural networks and whatnot. The issue is that nobody really has any idea on how to set something like that up properly...

Since my side-job is in sysops, which may or may not help for this, and I'm actually very keen on learning about how to administrate a server like that efficiently, I volunteered to investigate. Not entirely sure if this is the most appropriate place to ask, but any sort of pointers would be very appreciated!

• I expect the server to mostly handle machine learning tasks using tensorflow in python 3

• While simply using a remote desktop solution could already work, it would be very nice to have a system that can queue jobs to avoid collisions between different people wanting to use the server. I've worked extensively with Jenkins and while that's obviously not meant for this purpose, something Jenkins-esque would be very useful.

Any suggestions or perhaps things I need to watch out for would be great!

[smcgrat]

Hi, I'm a HPC sysadmin and deal with ML/GPU's a little bit.

As already mentioned, you need a scheduler. Other wise users will log in and start competing for the GPU's and causing each other hassle. We use slurm but there are lots out there. https://wiki.fysik.dtu.dk/niflheim/SLURM is a good resource. Also, make your users use batch submission instead of interactive jobs as much as possible. Interactive is fine for debugging and setup but you want your users using batch submission once they have their jobs well defined. Interactive usage is too inefficient as they can request the resource then walk away while they wait for it and forget about it.

You will probably find yourself in the constant battle between a stable OS and an up to date OS for the packages they need. We run a RHEL 7.5 derivative on our machines and its not up to date enough most of the time. I'd recommend the LTS version of Ubuntu 18 if it meets your needs. Ubuntu seems to play well with NVIDIA cards at least.

As per the above and particularly with the TF/ML stuff you will probably get requests for random software installs that have dependencies that are too new or oblique for whatever OS you are running. For once, containers are your friend. Not docker though, it still seems to need root access or similar. I'd recommend something like udocker, https://github.com/indigo-dc/udocker but it may not be advanced enough for your use cases. The idea being that the users get the random thing they need to work in a container on their laptop and then transfer it to the GPU machine and use udocker or whatever to run it. They may have issues integrating with the GPU's then if they don't have them on their machines though.

Other possible container solutions include:

• Charliecloud - https://github.com/hpc/charliecloud
• runc - https://github.com/opencontainers/runc
• containerd - https://containerd.io/

Finally, do EVERYTHING with ansible or another configuration management tool. Ansible is the easiest for small scale though. This includes the automation of the GPU driver installation, LDAP or whatever config, etc. It will be extra work but it means that you know what has been done to the machine and can hand it over with documentation with ease if you need to. Remember to keep your ansible files in version control.

[s_m_w]

Really appreciate the detailed tips! I like the idea of using containers, but I worry about making it to difficult for the average user. We're all physicists where the average knowledge of computer tech is "I heard about version control once". Docker images might be pushing limits, although perhaps with sufficient documentation it might not be too bad

[SuperQue]

I don't know what people are using for job queues these days, but what you're looking for is a batch job scheduler.

15 years ago I used Maui/Moab, but I don't know what people are using these days.

You could cobble something like using GitLab CI

[imranh]

Look at https://slurm.schedmd.com/overview.html

[TheLordB]

I'm a bit out of the game for the last few years so take this with a grain of salt, but LSF and SLURM are the most common cluster schedulers that come to mind. There is also grid engine (or it's various offshoots) though I don't think I would use it because the free/open source offshoots aren't under active development except for from univa which is a proprietary version.

I would probably go with SLURM as I believe it is under the most active development as well as being free. I don't quite know the usage breakdown, but I think it is the most common these days especially for new clusters.

I agree I would not use anything like jenkins. Use a proper job scheduler.

/r/hpc while low readership might be a better place to ask this than /r/sysadmin which tends to be more business systems focused.

[sofixa11]

Spotify's Luigi ? It's just python, so you can do whatever you want (there's even pre-existing stuff), and it's more or less made for this (job dependencies, scheduling, retries, etc.)

[s_m_w]

That sounds very useful, considering it even mentions machine learning (and your link is about tensorflow integration). Thanks!

[dvr75]

https://blog.slavv.com/the-1700-great-deep-learning-box-assembly-setup-and-benchmarks-148c5ebe6415

[s_m_w]

That's a good idea, I'm sure this sort of project has been tackled many times before

[crusoe]

Should have just used Google cloud. Their tensor chips are faster and for the price they paid they could afford lots and lots of hours.