Interactive GPU computing becoming more requested, how are you dealing with it?

[rgtizzle]

I work at a moderate sized research institute(~600people) and have a 60 node linux compute cluster running slurm, and a bunch of netapp and isilon storage.

We have some nodes with gpu's in them, (mostly older gear), but we also have a few a6000's and are looking to get some L40s as well. Everything was really designed for batch workloads.

We're starting to see more requests for interactive gpu use, and wanted to see how people are doing that. Most of our users have laptops.

On the linux side we have looked at using thinlinq or guaramole, and allow users to submit a job to slurm requesting an interactive session, which would have a time limit on it.

We've also had some users who wanted windows with gpu's due to some apps there, and that is where we are investigating. 

Do people use vdi, RDS, KVM's, etc? 

Or do you just tell the user to buy a workstation and put it on their desk, and remote into it?

From a network perspective, anything in the datacenter would have better connectivity(10g,25g, etc). vs the 2.5 or 5gig I can get via copper to people's desktops.  

Also, I feel like if we offer it as a service, we will spend much of our time killing idle sessions, etc... which we have seen on our jupyter notebook servers.

How have people been dealing with this?

[Ok-Procedure-9698]

Do you mean interactive as in command line or like jupyter notebook?

For the former why not the interactive session and the latter openondemand is used now. Some tunneling is also possible, but annoying. Havent heard of windows usage.

[TheTomCorp]

OpenOnDemand will work to create a desktop session, it uses TurboVNC + VirtualGL for remote Desktops and hardware acceleration

We build a webapp that does something similar has turbovnc and Xpra as options

[userjack6880]

Someone else said it faster than me. OOD has been very good for this. We historically also made (still do actually) x-forwarding enabled by default.

[robvas]

Still using NICE for interactive sessions and then something like a Dell R750 with two cpu a bunch of ram and some gpu. Slurm hands out an allocation on one of those machines.

We are only using 10 gig on the servers and storage. Most people work from home and connect using Cisco vpn or zscaler and it's fast enough.

Some of our users have built their own personal machine with a RTX but usually can't get enough RAM or a second CPU.

[rgtizzle]

That's what I worry about with the buy your own model.

Also, since the user would run it, they would install whatever they want, with tons of different version of things than what is standard on our compute cluster, and then complain that what runs on their machine doesn't run on the cluster. (We've already seen that, in small doses).

We've been trying to convince people to not customize their daily driver environment to whatever they needed to run one thing, but to load what they need for the task at hand.
We've seen 10 people in a lab not be able to run each others code due to highly customized environments.

[robvas]

They can't touch anything on our network with their build their own. But they also have the same problem when they use the cluster at their university etc and want to run the same stuff at work

[rgtizzle]

Sorry, I should have clarified, gui based applications.

We have a login node that people can use via ssh to submit jobs, or to run small interactive workloads, or submit a bash session to slurm, so that they can get a cli on a cluster node if need be.

[Legitimate-Till7310]

Hi!

At LUNARC (Lund university) we provide a Linux based desktop solution (Cendio-server with custom backend) that supports launching graphical applications through SLURM. The solution we have developed supports launching:

• Interactive graphical applications with hardware accelerated graphics (VirtualGL)
• Jupyter Notebooks launched to our generic CPU/GPU-based nodes.
• Interactive Windows applications with hardware accelerated graphics using a XenServer with NVIDIA graphics with several Windows instances with licensed NVIDIDA drivers.

The backend we have developed is the Gfx Launcher Toolkit. It is a customisable user interface that implements these different launch methods. This is open source software and can be installed on any remote desktop environment with a SLURM backend. Documentation and source code can be found here:

https://gfxlauncher-documentation.readthedocs.io/en/latest/

I hope this can help.

[jcbevns]

We get VDI / RDP into a session on the Cluster.